简体繁体 English

ADF 翻滚窗口触发器 - 自依赖

[英]ADF Tumbling window trigger - Self Dependency

原文 2021-07-08 03:56:07 6 1 azure/ azure-data-factory

I have created a tumbling window trigger with recurrence as 1 hour and attached it to a pipeline.我创建了一个重复为 1 小时的翻滚窗口触发器并将其附加到管道上。 Whenever the pipeline runs I want to process previous 2 hours after which I want to process current hour.每当管道运行时，我想处理前 2 小时，之后我想处理当前小时。

For Ex: If the pipeline is suppose to process records between 10 AM to 11 AM in that given run I want 8 AM to 10 AM records also to be processed after which I want current window 10 AM to 11 AM to be processed.例如：如果管道假设在给定的运行中处理上午 10 点到上午 11 点之间的记录，我希望还处理上午 8 点到上午 10 点的记录，之后我希望处理当前窗口上午 10 点到上午 11 点。 Similarly when 11 AM to 12 PM window kicks in I want 9 AM to 11 AM to be processed before processing 11 AM to 12 PM.同样，当上午 11 点到下午 12 点的窗口启动时，我希望在处理上午 11 点到下午 12 点之前处理上午 9 点到上午 11 点。

To achieve this functionality I have created a self dependency with offset as -0.02:00:00 and window size as 0.02:00:00.为了实现此功能，我创建了一个自依赖项，偏移量为 -0.02:00:00，窗口大小为 0.02:00:00。

But when pipeline is getting triggered it is looking for previous dependency windows (8 AM to 10 AM) from previous runs and only 10 AM to 11 AM records are getting processed.但是当管道被触发时，它会从之前的运行中寻找之前的依赖窗口（上午 8 点到上午 10 点），并且只有上午 10 点到上午 11 点的记录得到处理。

But my requirement is to even process 8 AM to 10 AM records along with 10 AM to 11 AM in that particular run.但我的要求是在该特定运行中甚至处理上午 8 点到 10 点的记录以及上午 10 点到 11 点的记录。

Is the functionality of the self dependency tumbling window limited to check only previous runs but not processing those records?自依赖滚动窗口的功能是否仅限于检查以前的运行但不处理这些记录？ Or am I missing something?或者我错过了什么？ How can above scenario be achieved via tumbling window trigger?如何通过翻转窗口触发器实现上述场景？ Is it possible to subtract 2 hours from @trigger().outputs.windowStartTime?是否可以从@trigger().outputs.windowStartTime 中减去 2 小时？

Below is the screenshot of the trigger.下面是触发器的截图。

TumblingWindowWithSelfDependency TumblingWindowWithSelfDependency

Any help is appreciated.任何帮助表示赞赏。 Thanks!谢谢！

1 个解决方案

While adding trigger on your pipeline, you should pass trigger parameters in to pipeline parameters, your pipeline should have two paramters like startTime and endTime to take values from trigger parameters在管道上添加触发器时，您应该将触发器参数传递给管道参数，您的管道应该有两个参数，例如 startTime 和 endTime，以从触发器参数中获取值

System parameters of tumbling window trigger:翻滚窗口触发器的系统参数：

trigger().outputs.windowStartTime --> 10AM - 2hrs == 8AM// it will process 8AM to 10AM records and the will process records between 10 AM to 11 AM trigger().outputs.windowStartTime --> 10AM - 2hrs == 8AM// 它将处理上午 8 点至上午 10 点的记录，并将处理上午 10 点至上午 11 点之间的记录

trigger().outputs.windowEndTime ---> 11AM - 2// Here there is no need of going for -2 hrs trigger().outputs.windowEndTime ---> 11AM - 2// 这里不需要 -2 小时

startTime --> trigger start time - 2 endTime --> Trigger end time startTime --> 触发开始时间 - 2 endTime --> 触发结束时间

There is no need of seeting the offset or dependencies.无需查看偏移量或依赖项。

To achive -2 hours use addhours() function: addHours()要达到 -2 小时，请使用addhours()函数： addHours()