Intermittent Azure Blob Upload Failures - 412 ConditionNotMet

Question

We are seeing intermittent errors when writing blobs to Azure Storage, and I'm having trouble isolating the possible cause.

This is the error we get:

Azure.RequestFailedException: The condition specified using HTTP conditional header(s) is not met.
RequestId:blahblahblah
Time:2021-08-07T07:39:08.0436441Z
Status: 412 (The condition specified using HTTP conditional header(s) is not met.)
ErrorCode: ConditionNotMet

The code where the exception is thrown looks roughly like this:

public async Task WriteBlobToStreamAsync(string blobName, Func<Stream, Task> asyncStreamingFunc)
{
    var blockBlobClient = _blobContainerClient.GetBlockBlobClient(blobName);
    await using var stream = await blockBlobClient.OpenWriteAsync(true);
    await asyncStreamingFunc(stream);
}

We're not setting any "conditional headers" in the request (unless something is happening under the covers that I'm not aware of).

The errors tend to occur during heavy load - we may have 20 or more tasks executing this code in parallel on a single instance, all writing to different blobs, in a Kube.netes cluster of 50 to 100 instances. As mentioned, this is a very intermittent error, but when it does occur, it seems to occur across all parallel tasks and some number of instances in the cluster.

How can I troubleshoot or at least attempt to narrow down the specific cause of this error? I've researched the Http status code and ErrorCode values, but can't find any specific guidance that would appear to apply to this case.

Answer 1

Possible work arounds that might help:

1) It might be If-Match eTag condition failure. Presence of etag in if-match header instructs storage service to perform the operation only if the condition matches (ie etag matches) specify an access condition with "*" value for if-match which essentially tells storage service to ignore the etag value.

var accessCondition = new AccessCondition();
var blobRequestOptions = new BlobRequestOptions();

var operationContext = new OperationContext();

// ETag access condition is used so that it will not cause any issue due to ongoing concurrent modification on the same blob

accessCondition.IfMatchETag = "*";

var blockBlobClient = _blobContainerClient.GetBlockBlobClient(blobName);
await using var stream = await blockBlobClient.OpenWriteAsync(accessCondition,blobRequestOptions, operationContext);
await asyncStreamingFunc(stream);

also see Example 5 from examples-access condition

For more information see about conditional headers and Managing Concurrency

(or) try

2) Try using the blob.UploadFromStreamAsync() method instead of blob.OpenWriteAsync().

Reference .

blob.OpenWriteAsync() is going to upload files in chunks as you write to the stream. This will be a problem if you're uploading larger files or a lot of files in parallel.