elasticsearch get date range of most recent ingestion

Question

I have an elasticsearch index that gets new data in large dumps, so from looking at the graph its very obvious when new data is added.

If I only want to get data from the most recent ingestion (in this case data from 2020-08-06, whats the best way of doing this?

I can use this query to get the most recent document:

GET /indexname/_search
{ 
      "query": {
        "bool": {
          "must": [
            {
              "query_string": {
                "query": queryString
              }
            }
          ]
        }
      },
      "sort": { 
        "@timestamp" : "desc" 
      }, 
      "size": 1 
    }

Which will return the most recent document, in this case a document with a timestamp of 2020-08-06. I can set that to my endDate and set my startDate to that date minus one day, but im worried of cases where the data was ingested overnight and spanned two days.

I could keep making requests to go back in time 5 hours at a time to find when the most recent large gap is, but im worried that making a request in a for loop could be time consuming? Is there a smarter way for getting the date range of my most recent ingestion?thx

Answer 1

When your data is coming in batches it'd be best to attribute an identifier to each batch. That way, there's no date math required.