Search API for ElasticSearch not returning all the logs

Question

I am wanting to get all the logs from elasticsearch from a asp.net core web api application using NEST. I created a controller named ESController to get the logs that are in elasticsearch. However, when I do so, on swagger it only displays the first log and not the entire list of logs. I even wrote to the console to see if I can view those logs there as well but it does not display. I do not have a model class for the logs. Is there something wrong I am doing to retrieve/extract the logs from elasticsearch in the .net application?

Console displays: "Valid NEST response built from a successful (200) low level call on POST: /elastic-search-app-logs%2A/_search?typed_keys=true"

swagger:

Correct me if I am wrong, but in the ElasticSearch CLI running the command below will display the logs from elasticsearch:

GET elastic-search-app-logs*/_search
{
  "query": {
    "match_all": {}
  }
}

So what I am trying to do is translate this code into .net with NEST.

The logs I am wanting to read/extract from elasticsearch into a .net application

Any pointers/suggestions would be greatly appreciated!!

Answer 1

The _search API is limited in the number of results it will return, even with a match_all filter.

The traditional way to get all results from Elasticsearch is scroll search . More recently the recommended approach is to use search_after instead (see the scroll search link for more info).

Docs for the NEST client cover both search_after and scrolling .

The _search API is limited to 10,000 results by default. While it's not recommended, you can change the limit with the index.max_result_window index setting .

[edit: typo fix]