[英]Download blobs from azure storage asynchronously and save them in DataTable
以下代碼顯示了如何從azure blob存儲區下載blob並將其保存到DataTable中:
foreach (var currIndexGroup in blobsGroupedByIndex)
{
DataRow dr = dtResult.NewRow();
foreach (var currIndex in currIndexGroup)
{
long fileByteLength = currIndex.Properties.Length;
byte[] serializedAndCompressedResult = new byte[fileByteLength];
currIndex.DownloadToByteArray(serializedAndCompressedResult, 0);
dr[currIndex.Metadata["columnName"]] = DeflateStream.UncompressString(serializedAndCompressedResult);
}
dtResult.Rows.Add(dr);
}
問題是下載速度很慢。 1000個真正的小Blob大約需要20秒才能下載。 如果嘗試使用currIndex.DownloadToByteArrayAsync(serializedAndCompressedResult, 0);
異步運行它currIndex.DownloadToByteArrayAsync(serializedAndCompressedResult, 0);
后續行將引發異常Bad state (invalid stored block lengths)
。
異步填充此數據表的正確方法是什么?
//the plan here is to make a model that holds your currIndex and byte array so you can return that model from a task
public class MyModel
{
public CloudBlockBlob CurrIndex {get;set;}
public byte[] FileBytes {get;set;}
}
foreach (var currIndexGroup in blobsGroupedByIndex)
{
var myTasks = new List<Task<MyModel>>();
foreach (var currIndex in currIndexGroup)
{
myTasks.Add(Task<MyModel>.Factory.StartNew(() =>
{
var myModel = new MyModel();
myModel.CurrIndex = currIndex;
long fileByteLength = myModel.CurrIndex.Properties.Length;
myModel.FileBytes = new byte[fileByteLength];
currIndex.DownloadToByteArray(myModel.FileBytes, 0);
return myModel;
});
}
Task.WaitAll(myTasks.ToArray());
foreach (var task in myTasks)
{
MyModel myModel = task.Result;
DataRow dr = dtResult.NewRow();
dr[myModel.CurrIndex.Metadata["columnName"]] = DeflateStream.UncompressString(myModel.FileBytes);
dtResult.Rows.Add(dr);
}
}
您可以通過在外部foreach循環上使用Parallel.ForEach
來進一步提高並行度。 您必須鎖定dtResult
使其線程安全。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.