Elasticsearch 429 Too Many Requests _bulk with synchronous requests

Question

I am using AWS Elasticsearch service. On dev environment there is t3.small instance. I have approx 15 000 records that I want to index as a bulk. What I do is splitting this amount on chunks of 250 items each (or lower than 10 MiB). And run _bulk request with refresh="wait_for" option, one by one, and waiting until request is finished before sending the next one. At some point, approximately on 25 iteration, the request is immediately fails with message

429 Too Many Requests /_bulk

Just in case, if chunk size will be 500 this will fail on 25/2 request (around 12)

It doesn't tell anything more. Just only this, I cannot understand why this happens if there is no anything else that could send bulk requests in parallel with me. I checked that the data size is lesser than 10MB.

What I already have

1. I send each request consistently, awaiting the previous one
2. Bulk request size is lesser than 10MiB
3. Each bulk request contains no more than 250 records in it (+ 250 to indicate that this is indexing)
4. I am using refresh="wait_for"
5. And even have 2 seconds delay before sending a new request (which I want strongly to remove)
6. Adding new instances or increasing storage space doesn't help at all

What could be the reason of having that error? How can I be guaranteed that my request will not be failed if I send everything consistently? Is there any additional option I can pass to be sure that bulk request is completely finished?

Answer 1

A 429 error message as a write rejection indicates a bulk queue error. The es_rejected_execution_exception[bulk] indicates that your queue is full and that any new requests are rejected. When the number of requests to the Elasticsearch cluster exceeds the bulk queue size (threadpool.bulk.queue_size), this bulk queue error occurs. A bulk queue on each node can hold between 50 and 200 requests, depending on which Elasticsearch version you are using.

You can consult this link https://aws.amazon.com/premiumsupport/knowledge-center/resolve-429-error-es/ and check the write rejection best practices