Get more than 1000 datasets from an Azure Table Storage

Question

I've an Azure Function to perform some operations on the datasets in an Azure Table Storage.

Since grouping is not working in Azure Table Storages, I have to fetch all datasets in my table and perform the operations I want (grouping, filtering) in my C# code.

But each query retrieves only the top 1000 datasets. How can I get all my datasets - or iterating over the table in bulks of 1000 to get all datasets at the end?

TableQuery<Models.product_item> query = new TableQuery<Models.product_item>()
          .Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, myPartitionKey));

var queryResult = myTable.ExecuteQuery(query);

Answer 1

When your query did not complete within certain limits (max. 5 seconds, max. 1000 rows, see here ), you'll receive a ContinuationToken in your result object. Pass this token to another query to continue your first query and get the next bunch of rows.

This extension method does the job for you:

public static class QueryExtensions
{
    public static async Task<IEnumerable<TElement>> ExecuteQueryAllElementsAsync<TElement>(this CloudTable table, TableQuery<TElement> tableQuery)
        where TElement : ITableEntity, new()
    {
        TableContinuationToken continuationToken = default(TableContinuationToken);
        var results = new List<TElement>(0);
        tableQuery.TakeCount = 500;

        do
        {
            //Execute the next query segment async.
            var queryResult = await table.ExecuteQuerySegmentedAsync(tableQuery, continuationToken);

            //Set exact results list capacity with result count.
            results.Capacity += queryResult.Results.Count;
            results.AddRange(queryResult.Results);

            continuationToken = queryResult.ContinuationToken;

        } while (continuationToken != null);

        return results;
    }
}

Then you can use it in your code like

var queryResult = await myTable.ExecuteQueryAllElementsAsync(query);

Answer 2

You can't. Probably to avoid long running query.

There are other limits wich you can read here: https://docs.microsoft.com/en-us/rest/api/storageservices/Query-Timeout-and-Pagination?redirectedfrom=MSDN