Writing CSV from Elasticsearch result using python with records exceeding 10000 ?

Question

Im able to create the CSV using the solution provided here:

Export Elasticsearch results into a CSV file

but problem arises when the records exceeds 10000 (size=10000), is there any way to write all the records?

Answer 1

The method you given in your question use elasticsearch's Python API, and es.search do have a 10 thousand docs retrieving limit.

If you want to retrieve data more than 10,000, as suggested by dshockley in the comment, you can try scroll API. Or you can try elasticsearch's scan helpers , which automates a lot work with scroll API. For example, you won't need to get a scroll_id and pass it to the API, which will be necessary if you use scroll directly.

When use helpers.scan, you need to specify index and doc_type in the parameters when call the function, or write them in the query body. Note that, the parameter name is 'query' rather than 'body'.