Azure Data Factory v2 - wrong year copying from parquet to SQL DB

Question

I'm having a weird issue with Azure Data Factory v2. There's a Spark Job which is running and producing parquet files as output, an ADFv2 copy activity then takes the output parquet and copies the data into an Azure SQL Database. All is working fine except for dates! When the data lands in SQL the year is 1969 years out. So todays date (2018-11-22) would land as 3987-11-22.

I've tried changing the source and destination types between Date, DateTime, DateTimeOffset and String but with no success. At the moment I'm correcting the dates in the database but this is not really ideal.

I've opened the source parquet files using Parquet Viewer, Spark and Python (desktop) and they all correctly show the year as 2018

Answer 1

According to parquet date type definition, https://drill.apache.org/docs/parquet-format/#sql-types-to-parquet-logical-types The date is stored as "the number of days from the Unix epoch, 1 January 1970 "

And ADF is using .net type doing the transformation. According to .net type definition, Time values are measured in 100-nanosecond units called ticks. A particular date is the number of ticks since 12:00 midnight, January 1, 000 1 AD (CE) https://docs.microsoft.com/en-us/dotnet/api/system.datetime?view=netframework-4.7.2

Seems extra 1969 is added for this reason. But not sure whether is this a bug. What is your parquet data type? is it Date? and what is the sql data type? Could you provide the copy activity run id? Or maybe some parquet sample data?

Answer 2

Based on Parquet encoding definitions ,no Date, DateTime, DateTimeOffset and String format exist,so you do not need to try with these formats.

Based on this Data type mapping for Parquet files in Azure Data Factory:

The DateTimeOffset format corresponds to Int96 ,I suggest you trying this transmission on the source of parquet file.