How to copy data from parquet source to delta destination in ADF

Question

How to copy data from parquet source to delta destination in Azure data factory

Answer 1

Create Linked services for both Parquet source and destination in the Azure Data Factory. Then create datasets for both source and sink to get their corresponding schemas. Then use a Copy Data activity and provide the necessary source and sink configurations in the corresponding fields of Copy data activity. Validate the pipeline and trigger the debug run of pipeline. Verify the data that was migrated to the destination.

Answer 2

Before copying the data, you need to check if your source is supported in Azure Data Factory or not.

Here is the list of all the Supported data stores in ADF .

You need to use Copy activity to copy the parquet file to the delta destination.

Apart from Copy activity , you will be using Linked Service to source and destination connection establishment. Then these Linked services will be used to define the Dataset of source (parquet) and destination (delta).

If your destination is Azure Databricks Delta Lake, you can refer the Microsoft Official document - Copy data to and from Azure Databricks Delta Lake using Azure Data Factory or Azure Synapse Analytics .