Azure DocumentDB Throttled Requests

Question

I have a document db database on azure. I have a particularly heavy query that happens when I archive a user record and all of their data.

I was on the S1 plan and would get an exception that indicated I was hitting the limit of RU/s. The S1 plan has 250.

I decided to switch to the Standard plan that lets you set the RU/s and pay for it.

I set it to 500 RU/s.

I did the same query and went back and looked at the monitoring chart.

At the time I did this latest query test it said I did 226 requests and 10 were throttled.

Why is that? I set it to 500 RU/s. The query had failed, by the way.

Answer 1

Firstly, Requests != Request Units, so your 226 requests will at some point have caused more than 500 Request Units to be needed within one second.

The DocumentDb API will tell you how many RUs each request costs, so you can examine that client side to find out which request is causing the problem. From my experience, even a simple by-id request often cost at least a few RUs. How you see that cost is dependent on which client-side SDK you use. In my code, I have added something to automatically log all requests that cost more than 10 RUs, just so I know and can take action.

It's also the case that the monitoring tools in the portal are quite inadequate and I know the team are working on that; you can only see the total RUs for every five minute interval, but you may try to use 600 RUs in one second and you can't really see that in the portal.

In your case, you may either have a single big query that just costs more than 500 RU - the logging will tell you. In that case, look at the generated SQL to see why, maybe even post it here.

Alternatively, it may be the cumulative effect of lots of small requests being fired off in a small time window. If you are doing 226 requests in response to one user action (and I don't know if you are) then you probably want to reconsider your design :)

Finally, you can retry failed requests. I'm not sure about other SDKs but the .Net SDK retries a request automatically 9 times before giving up (that might be another explanation for the 229 requests hitting the server). If your chosen SDK doesn't retry, you can easily do it yourself; the server will return a specific status code (I think 429 but can't quite remember) along with an instruction on how long to wait before retrying.

Please examine the queries and update your question so we can help further.