[英]Azure Data Factory: Dynamic path value for the Storage Event Trigger
I have created an azure data factory pipeline to copy the data from one adls container to another adls container using copy data activity.我创建了一个 azure 数据工厂管道,使用复制数据活动将数据从一个 adls 容器复制到另一个 adls 容器。 This copy activity will trigger using a storage event trigger.
此复制活动将使用存储事件触发器触发。
So whenever a new file gets generated, it will trigger the activity.因此,无论何时生成新文件,它都会触发该活动。
The source file is located in a nested directory structure having dynamic folders such as year, month, and day, which vary based on date.源文件位于一个嵌套的目录结构中,该目录结构具有动态文件夹,例如年、月和日,这些文件夹随日期而变化。
In the trigger, I mentioned the path until the fixed folder path, but I don't know what value I should put for the dynamic path.在触发器中,我提到了固定文件夹路径之前的路径,但我不知道应该为动态路径设置什么值。
Initially, I provided the path such as my/fixed/directory/*/*/*/
,最初,我提供了诸如
my/fixed/directory/*/*/*/
之类的路径,
but at the time of execution, it throws the exception 'PathNotFound'.但在执行时,它会抛出异常“PathNotFound”。
So my question is - How can I provide the path to the storage event trigger with the dynamic folder structure?所以我的问题是 - 如何使用动态文件夹结构提供存储事件触发器的路径? Following is ADF copy data pipeline screenshot:
以下是 ADF 复制数据管道屏幕截图:
Pipeline-管道-
Copy data activity source configuration-复制数据活动源配置-
Copy data activity target configuration-复制数据活动目标配置-
Copy data activity source dataset configuration-复制数据活动源数据集配置-
Copy data activity target dataset configuration-复制数据活动目标数据集配置-
blob path begins with
or blob path ends with
in storage event triggers.blob path begins with
或blob path ends with
为不支持通配符。input/folder/2022
is my fixed directory (input is container name).input/folder/2022
是我的固定目录(输入是容器名称)。 I also have sub folders within each of the folders shown below.folder path: @replace(dataset().folder_name,'input/','')
file name: @dataset().file_name
data
:data
的不同容器:folder path: @concat('output/',replace(dataset().folder,'input/folder/',''))
file name: @dataset().file
folderName
and fileName
will be set while creating trigger as shown below:folderName
和fileName
的值将在创建触发器时设置,如下所示:fileName : @triggerBody().fileName
folderName : @triggerBody().folderPath
folder/2022
the pipeline will be triggered.folder/2022
中的任何文件夹,管道就会被触发。folder/2022/03/01/sample1.csv
.folder/2022/03/01/sample1.csv
。 This triggered the pipeline successfully. So, creating a storage event trigger for just the parent directory is sufficient to be able to trigger the pipeline for any file uploaded to child directories as well.因此,仅为父目录创建存储事件触发器就足以触发上传到子目录的任何文件的管道。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.