CosmosDb - A write operation resulted in an error. Error=16500

Question

I have a database using MongoDB on CosmosDB.

Here is my retry pattern using Polly :

_retryPolicy = Policy
    .Handle<MongoCommandException>(e =>
    {
        if (e.Code != 16500 /*(RateLimitCode)*/ || !(e.Result is BsonDocument bsonDocument))
        {
            return false;
        }

        if (bsonDocument.TryGetValue("StatusCode", out var statusCode) && statusCode.IsInt32)
        {
            switch (statusCode.AsInt32)
            {
                case 429: //HttpThrottleErrorCode
                case 1: //HttpServiceIsUnavailable
                case 50: //HttpOperationExceededTimeLimit:
                    return true;
                default:
                    return false;
            }
        }

        return true;
    })
    .Or<MongoConnectionException>()
    .WaitAndRetryAsync(2, i => TimeSpan.FromSeconds(MongoRepositoryConstants.RETRY_POLICY_TIME_OUT_IN_SECOND));

And here is the code used to perform an UpdateMany with the MongoDb driver in C# :

public async Task<bool> UpdateManyAsync(IEnumerable<JObject> listRelatedQuotes, DateTime datetime, string quoteStatus)
{
    var listQuoteNumber = new BsonArray(listRelatedQuotes.Select(quote => quote[StdJsonDataLabel.toto][StdJsonDataLabel.QUOTE_IDENTIFIER_LABEL].ToString()));
    FilterDefinition<BsonDocument> filter = Builders<BsonDocument>.Filter.In(StdJsonDataPath.toto, listQuoteNumber);

    var update = Builders<BsonDocument>.Update.Set(StdJsonDataPath.fooooo, datetime.ToString("o"));

    if (!string.IsNullOrEmpty(quoteStatus))
    {
        update = update.Set(StdJsonDataPath.foooo2, quoteStatus);
    }

    bool res = false;
    await _retryPolicy.ExecuteAsync(async () =>
    {
        var result = await _collection.UpdateManyAsync(filter, update).ConfigureAwait(false);
        res = (result.MatchedCount > 0);
    });
    return res;
}

Unfortunaltely I get the following issue on CosmosDb :

A write operation resulted in an error. Error=16500, RetryAfterMs=12, Details=' A bulk write operation resulted in one or more errors. Error=16500, RetryAfterMs=12, Details='
MongoDB.Driver.MongoBulkWriteException`1[[MongoDB.Bson.BsonDocument, MongoDB.Bson, Version=2.8.1.0, Culture=neutral, PublicKeyToken=null]]

Following this documentation I know that the 16500 error code is an issue with the RU/sec on the database. But it is a MongoBulkWriteException so I'm wondering if it is handled by the retry policy

Following this documentation, the MongoBulkWriteException doesn't inherit from MongoCommandException . So can you please confirm that the Polly retry policy doesn't apply in this case?

EDIT : watching the CosmosDb dashboard on Azure, it looks like the UpdateMany costs a lot of RU :

Answer 1

We are currently running a private preview for a new feature for server-side retries for MongoDB users and I think this would benefit what you are doing here.

How this works is when 429's are encountered we will automatically retry requests up to 60 seconds before returning to the user. Our testing indicates this resolves nearly all of the issues customers see when doing bulk ingestion with MongoDB clients as you are or using tools like mongoimport.

If you are interested in participating in this private preview, please DM me on my Twitter profile markjbrown with your email address and Cosmos DB account names you would like for us to enable this feature for.

Thanks.

Answer 2

Answering to your question, yes, this Policy won't handle throttling errors during bulk writes.

You need to additionally handle MongoBulkWriteException . Than you should look into the contents of this exception, iterate over WriteErrors collection and check if specific element errored because of throttling and schedule it for a retry(ideally, compose a new bulk write collection from such failed records).