[英]How to capture an output value in Copy Activity in Azure Data Factory
I'm fetching data from an API source, which with each pagination is giving back some meta data needed to control the data flow.我正在从 API 来源获取数据,每次分页都会返回一些控制数据流所需的元数据。 Most importantly it gives me the next page URL (captured in the "pagination rules") and the [end_of_stream] value.
最重要的是,它给了我下一页 URL(在“分页规则”中捕获)和 [end_of_stream] 值。
However I am failing to understand how I can capture this output value and encapsulate the Copy Activity in a, for example, While-loop (While [end_of_stream] = 0, continue fetching data).但是,我无法理解如何捕获此 output 值并将复制活动封装在 While 循环中(当 [end_of_stream] = 0 时,继续获取数据)。
Could someone please assist?有人可以帮忙吗? Do I need to use the Web Activity instead, as this allows me to view the output of the activity and use those values in variables.
我是否需要改用 Web 活动,因为这允许我查看活动的 output 并在变量中使用这些值。 The Copy Activity only seems to have the option of assigning values to columns in the sink.
复制活动似乎只能选择将值分配给接收器中的列。 And the output of said activity is only showing meta-data not usable for me.
而上述活动的 output 仅显示对我不可用的元数据。
I'll answer my own question, in case someone else is having the same headache.我会回答我自己的问题,以防其他人有同样的头痛。
What I did not realize is the built-in functionality of the Copy Activity, with the Pagination Rules ( reference ).我没有意识到的是复制活动的内置功能,以及分页规则( 参考)。
It contains the option of adding an end condition , which can be picked up by the Copy Activity with each iteration.它包含添加结束条件的选项,可以在每次迭代时由复制活动选取。 The iteration is also picking up the next page, via the next page URL found in each response from the API (in my case, the Zendesk API).
迭代还通过在 API(在我的例子中是 Zendesk API)的每个响应中找到的下一页 URL 获取下一页。
In my JSON, it looks like:在我的 JSON 中,它看起来像:
"EndCondition:$.end_of_stream": "Const:true"
What I would like to do is save the last picked up cursor as an variable, but I guess the best way to do this is by adding this as a column to my source on each row.我想做的是将最后一次拾取的 cursor 保存为一个变量,但我想最好的方法是将它作为一列添加到我的每一行的源代码中。
Pagination Rules for REST API with end condition: REST API 结束条件的分页规则:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.