DocumentDB return “Request rate is large”, parse on azure

Question

I'm runing parse on azure (Parse Server on managed Azure services), I'ts include DocumentDB as database and have limit for requests per seconds. Some parse cloud functions are large and the speed of requests is too high (even for S3 tier) so i'm getting this error (seen using Visual Studio Team Services (was Visual Studio Online) and Streaming logs).

    error: Uncaught internal server error. { [MongoError: Message: {"Errors":["Request rate is large"]}

ActivityId: a4f1e8eb-0000-0000-0000-000000000000, Request URI: rntbd://10.100.99.69:14000/apps/f8a35ed9-3dea-410f-a89a-28650ff41381/services/2d8e5320-89e6-4750-a06f-174c12013c69/partitions/53e8a085-9fed-4880-bd90-f6191765f625/replicas/131091039101528218s]
  name: 'MongoError',
  message: 'Message: {"Errors":["Request rate is large"]}\r\nActivityId: a4f1e8eb-0000-0000-0000-000000000000, Request URI: rntbd://10.100.99.69:14000/apps/f8a35ed9-3dea-410f-a89a-28650ff41381/services/2d8e5320-89e6-4750-a06f-174c12013c69/partitions/53e8a085-9fed-4880-bd90-f6191765f625/replicas/131091039101528218s' } MongoError: Message: {"Errors":["Request rate is large"]}

ActivityId: a4f1e8eb-0000-0000-0000-000000000000, Request URI: rntbd://10.100.99.69:14000/apps/f8a35ed9-3dea-410f-a89a-28650ff41381/services/2d8e5320-89e6-4750-a06f-174c12013c69/partitions/53e8a085-9fed-4880-bd90-f6191765f625/replicas/131091039101528218s
    at D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:673:34
    at handleCallback (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:159:5)
    at setCursorDeadAndNotified (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:501:3)
    at nextFunction (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:672:14)
    at D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:585:7
    at queryCallback (D:\home\site\wwwroot\node_modules\mongodb-core\lib\cursor.js:241:5)
    at Callbacks.emit (D:\home\site\wwwroot\node_modules\mongodb-core\lib\topologies\server.js:119:3)
    at null.messageHandler (D:\home\site\wwwroot\node_modules\mongodb-core\lib\topologies\server.js:397:23)
    at TLSSocket.<anonymous> (D:\home\site\wwwroot\node_modules\mongodb-core\lib\connection\connection.js:302:22)
    at emitOne (events.js:77:13)

How to handle this error?

Answer 1

TL;DR;

1. Upgrade the old S3 collection to a new single collection under the new pricing scheme. This can support up to 10K RU (up from 2500 RU)
2. Delete the old S3 collection and create a new partitioned collection. Will require support for partitioned collection in parse.
3. Implement a backoff strategy in line with the x-ms-retry-after-ms response header.

Long answer:

Each request to DocumentDB returns a HTTP header with the Request charge for that operation. The number of request units is configured per collection. As per my understanding you have 1 collection of size S3, so this collection can only handle 2500 Request Units per second.

DocumentDB scales by adding multiple collections. With the old configuration using S1 -> S3 you must do this manually, ie you must distribute your data over the collections using an algorithm such as consistent hashing, a map or perhapse date. With the new pricing in DocumentDB you can use partitioned collections, by defining a partition key, DocumentDB will shard your data for you. If you see sustained rates of RequestRateTooLarge errors I recommend scaling out the partitions. However, you will need to investigate if Parse supports partitined collections.

When you receive a HTTP 429 RequestRateTooLarge there's also a header called x-ms-retry-after-ms :### where ### denotes the number of milliseconds to wait before you retry the operation. What you can do is to implement a back-off strategy which retries the operation. Do note that if you have clients hanging on the server during retries, you may build up request queues and clog the server. I recommend adding a Queue to handle such burst. For short burst of requests this is a nice way to handled it without scaling up the collections.

Answer 2

i used Mlab as external mongoDB database and configure the parse app in azure to use it instead of documentDB.

I have to will to pay so much for "performance" increase.