Azure Data Factory foreach activity step size support

Question

I have a pipeline that contains a list of IDs as input and I need to iterate through these IDs and call a REST API using batches of 10 IDs per time (these IDs will be passed as a parameter into JSON request).

1) Is there any approach using forEach activity in Data Factory passing the step size?

2) Do you have any other suggestions of how to accomplish this?

I have tried using "forEach" loop and also thinking in a way to use "setVariable" and "appendVariable" activities to store the current index during the loop, but also couldn't find a way to get the current index during the "forEach".

Answer 1

You should use a LookupActivity . With that you can get information from database, files or whatever and them pass it to a ForEach Loop.

Consider I have the following information in my txt file:

name|age
orochiBrabo|25
NarutoBoy|98

You can recover it using LookupActivity which I will call MyLookUp and then connect it box with a ForEach Box.

In ForEach Activity setting tab you write @activity('MyLookUp').output.value and now you can iterate over all rows in the file. Inside your ForEach you can refer results like item().age , item().name or item().myColumnName .