Intermittent error while downloading blobs in Azure

Question

In my web app running on Azure, I seem to be getting these errors in the windows event logs roughly once 10 minutes while the site is under heavy load

The server encountered an unknown failure: The operation has timed out
Stack Trace:
at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()
at Microsoft.WindowsAzure.StorageClient.RequestWithRetry.RequestWithRetrySyncImpl[TResult](ShouldRetry retryOracle, SynchronousTask`1 syncTask)
at Microsoft.WindowsAzure.StorageClient.CloudBlob.DownloadByteArray(BlobRequestOptions options)

But its not all the time, just every now and then, I really dont know how to diagnose it or even find out why this could be occurring.

Can I provide you with some sample code? What would you need to see? Unfortunately, this app is under NDA so I cannot talk about specifics or give to much away.

I have cross posted this on the MS forums as well, and I will share all answers I get on any site. This is a real production issue at the moment and any help here is greatly appreciated.

Answer 1

You mentioned that these errors occur when under heavy load. There are a few factors that could result in intermittent errors against Storage:

• Network bandwidth. You get approx. 100Mbps per core. Could you be saturating the network bandwidth between compute and storage? I'm guessing you have multiple role instances, which would reduce the likelihood of hitting this particular issue.
• Per-item throughput and transaction rate. Each blob, effectively stored in its own partition, has a target of 60MB/s. Do you have multiple readers of the same objects? If so, it's possible you're hitting timeouts due to too many readers requesting the same object, either in throughput or in transaction count (each object has a target of 500 transactions / second).
• Per-account transaction rate. A storage account has an overall scale target of 5,000 transactions per second. Access enough blobs simultaneously, and you could potentially exceed this limit.

More details on scalability targets here .