在 Lookup @ Azure Data Factory 中处理 >5000 行

Question

I have a Copy Activity which copies a Table from MySQL to Azure Table Storage. This works great. But when I do a Lookup on the Azure Table I get an error. (Too much Data)

This is as designed referred to the documentation: The Lookup activity has a maximum of 5,000 rows, and a maximum size of 2 MB.

Also there is a Workaround mentioned: Design a two-level pipeline where the outer pipeline iterates over an inner pipeline, which retrieves data that doesn't exceed the maximum rows or size.

How can I do this? Is there a way to define a offset (eg only read 1000 rows)

Answer 1

Do you really need 5000 iterations of your foreach? What kind of process are you doing in the foreach, isn't there a more efficient way of doing that?

Otherwise, maybe the following solution might be possible.

Create a new pipeline with 2 integer variables: iterations and count with 0 as defaults.

First determine the needed number of iterations. Do a lookup to determine the total number of datasets. In your query divide this by 5000, add one and round it upwards. Set the value of the iterations variable to this value using the set variable activity.

Next, add a while loop with expression something like @less(variables('count'),variables('iterations')). in this while loop call your current pipeline and pass the count variable as a parameter. After the execute pipeline activity, set the count variable to +1.

In your current pipeline you can use the limit/offset clause in combination with the passed parameter in a MySQL query to get the first 0-5000 results for your first iteration, 5000-10000 for your second iteration etc..

If you really need to iterate on the table storage, the only solution I see is that you'll have to create pagination on the resultset yourself, you could use a logic app for this purpose and call it by using a webhook.

Answer 2

I have a Copy Activity which copies a Table from MySQL to Azure Table Storage. This works great. But when I do a Lookup on the Azure Table I get an error. (Too much Data)

This is as designed referred to the documentation: The Lookup activity has a maximum of 5,000 rows, and a maximum size of 2 MB.

Also there is a Workaround mentioned: Design a two-level pipeline where the outer pipeline iterates over an inner pipeline, which retrieves data that doesn't exceed the maximum rows or size.

How can I do this? Is there a way to define a offset (eg only read 1000 rows)