如何使用Azure Data Factory V2（ADF）在文件夹中查找最新文件

Question

I am trying to read the latest blob file(csv) using azure data factory v2. The file name also contains date(YYYY-MM-DD mm:ss-abcd.csv). I need to read the data from the latest file present and load into table storage. Could you please help me with how to read the latest file using ADF

Answer 1

Hello Faiz Rahman and thank you for your question. The date format you chose has the useful feature of lexicographic sorting matching chronological sorting. This means, once you have a list of blobs, extracting the date and comparing is all that is needed.

If you have a very large list of blobs, this might not be practical. In that case, whenever you write a new blob, keep track of it somewhere, say, "maxBlobName.txt", and have pipeline read that to get the name of the most recent file.

Here is some example code for comparing the date portion of your blob names. To adapt for your purposes, you will need to use GetMetadata activity to fetch the blob names, and some string functions to extract only the date portion of the name for comparison.

{
"name": "pipeline9",
"properties": {
    "activities": [
        {
            "name": "ForEach1",
            "type": "ForEach",
            "dependsOn": [
                {
                    "activity": "init array",
                    "dependencyConditions": [
                        "Succeeded"
                    ]
                }
            ],
            "typeProperties": {
                "items": {
                    "value": "@variables('list')",
                    "type": "Expression"
                },
                "isSequential": true,
                "activities": [
                    {
                        "name": "If Condition1",
                        "type": "IfCondition",
                        "typeProperties": {
                            "expression": {
                                "value": "@greater(item(),variables('max'))",
                                "type": "Expression"
                            },
                            "ifTrueActivities": [
                                {
                                    "name": "write new max",
                                    "type": "SetVariable",
                                    "typeProperties": {
                                        "variableName": "max",
                                        "value": {
                                            "value": "@item()",
                                            "type": "Expression"
                                        }
                                    }
                                }
                            ]
                        }
                    }
                ]
            }
        },
        {
            "name": "init array",
            "type": "SetVariable",
            "typeProperties": {
                "variableName": "list",
                "value": {
                    "value": "@split(pipeline().parameters.input,',')",
                    "type": "Expression"
                }
            }
        }
    ],
    "parameters": {
        "input": {
            "type": "string",
            "defaultValue": "'2019-07-25','2018-06-13','2019'-06-24','2019-08-08','2019-06-23'"
        }
    },
    "variables": {
        "max": {
            "type": "String",
            "defaultValue": "0001-01-01"
        },
        "list": {
            "type": "Array"
        }
    }
}

}