Azure Data Factory - Copying FTP recursively to database

Question

I am trying to copy csv files that are recursively put inside a FTP folder to a SQL database in Azure. I also have subscription to Azure Data Factory. Can some one help me with the easiest way to accomplish this ?

What I have tried

1. Used the copy utility inside the Azure portal. The problem with it is that, it is asking me to copy to a table defined in the database. Can it just read from the csv files and detect the schema and put it into tables directly ?

2. Tried to use .NET sdk. But I don't see an example for ftp to sql database in azure directly. The examples only tell how to store into a blob storage.

Answer 1

If the table in database already exists, you can use ADF to copy from FTP to Azure SQL. Reference ADF docs here https://docs.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal . Though it's a tutorial from Azure Blob to Azure SQL, you can set FTP as source and other steps are similiar.

But ADF can only copy to existing tables, if the table doesn't exist before, you will have to write a sql script to create the table before the copy. You can put the script at pipeline-copyactivity-sink-"Pre-copy script". It will be ran before copy starts.

Answer 2

Can it just read from the csv files and detect the schema and put it into tables directly ?

Based on the sql database dataset in azure data factory, tableName property is required. So, if you need to transfer csv data into sql database, you need to create related tables so that they could establish a mapping relationship.

Tried to use .NET sdk. But I don't see an example for ftp to sql database in azure directly. The examples only tell how to store into a blob storage.

Based on your requirements, I suggest you follow two steps.

Step 1 : Create a copy activity from ftp to azure blob storage .

Step 2: Create another copy activity from azure blob storage to sql database .

You can combine two activity into one flow.

Hope it helps you,any concern please let me know.

---

Update answer:

You still need to copy the csv files from ftp into azure blob storage.If you have 2 target tables,just create two containers in blob storage so that you could divide the files into the specific container.

Then just configure the container name in azure blob storage dataset.

It will loop all the files in the container.