转换 Azure.AsyncPageable<blobitem> 列出<blobitem></blobitem></blobitem>

Question

I am retrieving lakhs of blobs from Azure blob storage container. Based on the retrieval I am calculating the retrieval time and need to convert the blobs as readable format and need to get the blob items count in the result. Hence I am retrieving the blobs and converting it into List< BlobItem > as mentioned below. Now the problem is, when I retrieve all the blobs available in the container using the method GetBlobsAsync() , it is taking few milliseconds to get all the blob items from the storage container. But when am converting the blob items into List< BlobItem > using blobItems.ToListAsync() , it is taking 5-10 minutes to complete it.

var watch = System.Diagnostics.Stopwatch.StartNew();
BlobContainerClient client = new BlobContainerClient(connectionString, containerName);
var blobItems =  client.GetBlobsAsync(); //This line take 60-80 milli seconds to get all blobs
var blobList =await blobItems.ToListAsync(); //This line takes 5-10mins to complete
watch.Stop();
var elapsedMs = watch.ElapsedMilliseconds;
return new OkObjectResult(blobList);

During the conversion only it is taking too long to complete all the blobs. Even I tried the same with Foreach loop, it is also taking long time(5-10 mins) to fetch the entire data as readable data. Need help on converting Azure.AsynPageable to some other readable format within some milliseconds. If there will be any other ways, we can achieve the same also fine. Thanks

Answer 1

Now the problem is, when I retrieve all the blobs available in the container using the method GetBlobsAsync(), it is taking few milliseconds to get all the blob items from the storage container. But when am converting the blob items into List< BlobItem > using blobItems.ToListAsync(), it is taking 5-10 minutes to complete it.

This is not what is happening. GetBlobsAsync returns an AsyncPageable . It does not actually fetch the blob items. That only happens when you iterate over the result, as stated in the docs

The GetBlobsAsync(BlobTraits, BlobStates, String, CancellationToken) operation returns an async sequence of blobs in this container. Enumerating the blobs may make multiple requests to the service while fetching all the values.

In your case, the code await blobItems.ToListAsync(); enumerates the blobs so that will take most of the time.

Got your point. But await blobItems.ToListAsync(); is taking long time to process the data retrieval. Is there any way to speedup the retrieve process by using any parallel programing process or any other way?

If you have a virtual folder structure you could fetch blob items in batches using the prefix parameter of the GetBlobsAsync method like this:

async Task DownloadBlobsAsync(BlobContainerClient client)
{
    var blobItemLists = await Task.WhenAll(new[] {
        DownloadBlobsByPrefix("folder1"),
        DownloadBlobsByPrefix("folder2")
    });
    
    var  blobItems = blobItemLists.SelectMany(items => items);
}

async Task<List<BlobItem>> DownloadBlobsByPrefix(string prefix)
{
    var blobItems = client.GetBlobsAsync(prefix: prefix);
    return await blobItems.ToListAsync();
}

but you will need to measure to see if this has the effect you want. Also, you need to think about the right set of prefixes to use.