How to capture an output value in Copy Activity in Azure Data Factory

Question

I'm fetching data from an API source, which with each pagination is giving back some meta data needed to control the data flow. Most importantly it gives me the next page URL (captured in the "pagination rules") and the [end_of_stream] value.

However I am failing to understand how I can capture this output value and encapsulate the Copy Activity in a, for example, While-loop (While [end_of_stream] = 0, continue fetching data).

Could someone please assist? Do I need to use the Web Activity instead, as this allows me to view the output of the activity and use those values in variables. The Copy Activity only seems to have the option of assigning values to columns in the sink. And the output of said activity is only showing meta-data not usable for me.

Data Factory Copy Activity

Answer 1

I'll answer my own question, in case someone else is having the same headache.

What I did not realize is the built-in functionality of the Copy Activity, with the Pagination Rules ( reference ).

It contains the option of adding an end condition , which can be picked up by the Copy Activity with each iteration. The iteration is also picking up the next page, via the next page URL found in each response from the API (in my case, the Zendesk API).

In my JSON, it looks like:

"EndCondition:$.end_of_stream": "Const:true"

What I would like to do is save the last picked up cursor as an variable, but I guess the best way to do this is by adding this as a column to my source on each row.

Pagination Rules for REST API with end condition: