Elasticsearch : retrieve all documents from index with python

Question

I need to retrieve documents from Elasticsearch in Python.

So I wrote this small code :

es = Elasticsearch(
            myHost,
            port=myPort,
            scheme="http")

request = '''{"query": {"match_all": {}}}'''

results = es.search(index=myIndex, body=request)['hits']['hits']

print(len(results))
>> 10

The problem is that it only returns 10 documents from my index when I expect to have few hundreds. How is it possible to retrieve all documents from the index ?

Answer 1

You have several ways to solve this.

If you know the maximum amount of documents you will have in the index, you can set the size parameter of the search as that number or more. For example, if you know you will have less than 100, you can retrieve them this way results = es.search(index=myIndex, body=request, size=100)['hits']['hits']

If you don't know that number, and you still want all of them, you will have to use the scan function, instead of the search function. The documentation for that is here