根据Azure数据工厂中的文件名将文件从一个文件夹复制到多个文件夹

Question

我在 ADLS Gen2 中有一个名为Source的父文件夹，它有许多子文件夹，这些子文件夹包含实际数据文件，如下例所示......

***来源： ***

文件夹名称： 20221212

A_20221212.txt B_20221212.txt C_20221212.txt

文件夹名称： 20221219

A_20221219.txt B_20221219.txt C_20221219.txt

文件夹名称： 20221226

A_20221226.txt B_20221226.txt C_20221226.txt

如何使用 Azure 数据工厂将文件从子文件夹复制到命名特定文件夹（如果不存在则应创建一个新文件夹），请参见下面的示例...

***目标： ***

文件夹名称： A

A_20221212.txt A_20221219.txt A_20221226.txt

文件夹名称： B

B_20221212.txt B_20221219.txt B_20221226.txt

文件夹名称： C

C_20221212.txt C_20221219.txt C_20221226.txt

非常感谢您的帮助。

Answer 1

我已经复制了上面的内容并得到了以下结果。

如果文件夹目录处于同一级别，则可以使用“获取元数据”活动执行以下过程。

这是我的源文件夹结构。

data
    20221212
        A_20221212.txt
        B_20221212.txt
        C_20221212.txt`
    20221219
        A_20221219.txt
        B_20221219.txt
        C_20221219.txt
    20221226
        A_20221226.txt
        B_20221226.txt
        C_20221226.txt

源数据集：

将其提供给 Get Meta 数据活动并使用ChildItems 。

然后将 ChildItems 数组从 Get Meta 数据活动提供给 ForEach 活动。 在 ForEach 内部，我使用 set 变量来存储文件夹名称。

@split(item().name,'_')[0]

现在，使用复制活动并在源代码中使用通配符路径，如下所示。

对于接收器，创建数据集参数并为其提供复制活动接收器，如下所示。

我的管道 JSON：

{
    "name": "pipeline1",
    "properties": {
        "activities": [
            {
                "name": "Get Metadata1",
                "type": "GetMetadata",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "dataset": {
                        "referenceName": "sourcetxt",
                        "type": "DatasetReference"
                    },
                    "fieldList": [
                        "childItems"
                    ],
                    "storeSettings": {
                        "type": "AzureBlobFSReadSettings",
                        "enablePartitionDiscovery": false
                    },
                    "formatSettings": {
                        "type": "DelimitedTextReadSettings"
                    }
                }
            },
            {
                "name": "ForEach1",
                "type": "ForEach",
                "dependsOn": [
                    {
                        "activity": "Get Metadata1",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "items": {
                        "value": "@activity('Get Metadata1').output.childItems",
                        "type": "Expression"
                    },
                    "isSequential": true,
                    "activities": [
                        {
                            "name": "Copy data1",
                            "type": "Copy",
                            "dependsOn": [
                                {
                                    "activity": "Set variable1",
                                    "dependencyConditions": [
                                        "Succeeded"
                                    ]
                                }
                            ],
                            "policy": {
                                "timeout": "0.12:00:00",
                                "retry": 0,
                                "retryIntervalInSeconds": 30,
                                "secureOutput": false,
                                "secureInput": false
                            },
                            "userProperties": [],
                            "typeProperties": {
                                "source": {
                                    "type": "DelimitedTextSource",
                                    "storeSettings": {
                                        "type": "AzureBlobFSReadSettings",
                                        "recursive": true,
                                        "wildcardFolderPath": "*",
                                        "wildcardFileName": {
                                            "value": "@item().name",
                                            "type": "Expression"
                                        },
                                        "enablePartitionDiscovery": false
                                    },
                                    "formatSettings": {
                                        "type": "DelimitedTextReadSettings"
                                    }
                                },
                                "sink": {
                                    "type": "DelimitedTextSink",
                                    "storeSettings": {
                                        "type": "AzureBlobFSWriteSettings"
                                    },
                                    "formatSettings": {
                                        "type": "DelimitedTextWriteSettings",
                                        "quoteAllText": true,
                                        "fileExtension": ".txt"
                                    }
                                },
                                "enableStaging": false,
                                "translator": {
                                    "type": "TabularTranslator",
                                    "typeConversion": true,
                                    "typeConversionSettings": {
                                        "allowDataTruncation": true,
                                        "treatBooleanAsNumber": false
                                    }
                                }
                            },
                            "inputs": [
                                {
                                    "referenceName": "sourcetxt",
                                    "type": "DatasetReference"
                                }
                            ],
                            "outputs": [
                                {
                                    "referenceName": "targettxts",
                                    "type": "DatasetReference",
                                    "parameters": {
                                        "folder_name": {
                                            "value": "@variables('folder_name')",
                                            "type": "Expression"
                                        },
                                        "file_name": {
                                            "value": "@item().name",
                                            "type": "Expression"
                                        }
                                    }
                                }
                            ]
                        },
                        {
                            "name": "Set variable1",
                            "type": "SetVariable",
                            "dependsOn": [],
                            "userProperties": [],
                            "typeProperties": {
                                "variableName": "folder_name",
                                "value": {
                                    "value": "@split(item().name,'_')[0]",
                                    "type": "Expression"
                                }
                            }
                        }
                    ]
                }
            }
        ],
        "variables": {
            "folder_name": {
                "type": "String"
            }
        },
        "annotations": []
    }
}

结果：