Azure 数据工厂在复制到 Blob 存储时指定自定义输出文件名

Question

I'm currently using ADF to copy files from an SFTP server to Blob Storage on a scheduled basis.

The filename structure is AAAAAA_BBBBBB_CCCCCC.txt.

Is it possible to rename the file before copying to Blob Storage so that I end up with a folder-like structure like below?

AAAAAA/BBBBBB/CCCCCC.txt

Answer 1

Here is what worked for me

I created 3 parameters in my Blob storage dataset, see the image bellow:

I specified the name of my file, added the file extension, you can add anything in the Timestamp just so you could bypass the ADF requirement since a parameter can't be empty.

Next, click on the Connection tab and add the following code in the FileName box: @concat(dataset().FileName,dataset().Timestamp,dataset().FileExtension). This code basically concatenate all parameters do you could have something like "FileName_Timestamp_FileExtension. See the image bellow:

Next, click on your pipeline then select your copy data activity. Click on the Sink tab. Find the parameter Timestamp under Dataset properties and add this code: @pipeline().TriggerTime. See the image bellow:

Finally, publish your pipeline and run/debug it. If it worked for me then I am sure it will work for you as well :)

Answer 2

With ADF V2, you could do that. First, use a lookup activity to get all the filenames of your source. Then chain a foreach activity to iterate the source file names. The foreach activity contains a copy activity. Both your source dataset and sink dataset of the cop activity have parameters for filename and folder path. You could use split and replace functions to generate the sink folder path and filename based on your source file names.

Answer 3

First you have to get the filenames in a GetMetadata-Activity. You can use this as a parameter in a copy-Activity and rename the filenames.

As mentioned in previous answer you can use a replace function to do this:

{
    "name": "TgtBooksBlob",
    "properties": {
        "linkedServiceName": {
            "referenceName": "Destination-BlobStorage-data",
            "type": "LinkedServiceReference"
        },
        "folder": {
            "name": "Target"
        },
        "type": "AzureBlob",
        "typeProperties": {
            "fileName": {
                "value": "@replace(item().name, '_', '\\')",
                "type": "Expression"
            },
            "folderPath": "data"
       }
    },
    "type": "Microsoft.DataFactory/factories/datasets"
}