Azure ADF V2 ForEach File CopyData from Blob Storage to SQL Table

Question

I need to design an ADF pipeline to copy a CSV file created on a particular Blob Store folder path named "Current" to a SQL table. After successful copy, i'll have to move the file to an archive folder.

Things i've accomplished : I'm using a Copy Data Activity that copies the CSV file and loads into my SQL table. On success, another copy data activity will copy the CSV file from "Current" folder to "Archive", after this i have a Delete activity that deletes the file from Current Folder.

Issue : This is totally fine where there is only one file at a time. But i want redesign this to handle multiple files imports. I want make use of ForEach activity. Please help.

Answer 1

You need to add a Get Metadata activity before the for-each. The Get Metadata activity will get the files in the current directory and pass them to the For-Each. You connect it to your Blob storage folder and specify the file mask. The output of the activity will need to be the input for your for-each loop. Something like this..@activity('Get Metadata').output. By the way how are you triggering this ADF pipeline? Another way to accomplish this would be to setup an event trigger to run this pipeline whenever a file is created in that blob directory. Here are the steps to use the For-Each on files in a storage container.

1. Set the Get Metadata argument to "Child Items"
2. In your For-Each set the Items to @activity('Get Metadata1').output.childitems

3. In the Source Dataset used in your Copy Activity create a parameter named FileName.

4. Set the file path to use the parameter

5. On the Copy Activity set the FileName parameter to @item().name

To finish you'll need to set the sink to whatever you need. That should do it.