数据工厂 | 从多个子文件夹递归复制到一个同名文件夹

Question

Objective: Copy all files from multiple subfolders into one folder with same filenames. Eg

Source Root Folder
20221110/
  AppID1
    File1.csv
    File2.csv
  /AppID2
     File3.csv
     File4.csv
20221114
   AppID3
     File5.csv
     File6.csv
and so on
Destination Root Folder
    File1.csv
    File2.csv
    File3.csv
    File4.csv
    File5.csv
    File6.csv

Approach 1 Azure Data Factory V2 All datasets selected as binary

1. GET METADATA - CHILDITEMS
2. FOR EACH - Childitem
3. COPY ACTIVITY(RECURSIVE: TRUE, COPY BEHAVIOUR: FLATTEN)

This config renames the files with autogenerated names. If I change the copy behaviour to preserve hierarchy, Both file name and folder structure remains intact.

Approach 2

1. GET METADATA - CHILDITEMS
2. FOR EACH - Childitems
3. Execute PL2 (Pipeline level parameter: @item.name)
4. Get Metadata2 (Parameterised from dataset, invoked at pipeline level)
5. For EACH2- Childitems
6. Copy (Source: FolderName - Pipeline level, File name - ForEach2)

Both approaches not giving the desired output. Any help/Workaround would be appreciated.

Answer 1

My understanding is in Option 2 Step 3 & 5:is done as to iterate through the folder and subfolder correct?

6. Copy (Source: FolderName - Pipeline level, File name - ForEach2)

I think since in step 6 you already have the filename. On the SINK side, add an dynamic expression and add @Filename and that should do the trick.

Answer 2

If all of your files are in the same directory level , you can try the below approach.

First use Get Meta data activity to get all files list and then use copy inside ForEach to copy to a target folder.

These are my source files with directory structure:

Source dataset:

Based on your directory level use the wildcard placeholder( */* ) in the source dataset.

The above error is only a warning, and we can ignore it while debug.

Get meta data activity:

This will give all the files list inside subfolders.

Give this array to a ForEach activity and inside ForEach use copy activity.

Copy activity source:

In the above also, the */* should be same as we gave in Get Meta data.

For sink dataset create a dataset parameter and use in the file path of dataset.

Copy activity sink:

Files copied to target folder:

If your source files are not in same directory level then you can try the recursive approach mentioned in this article by @Richard Swinbank .