How to fetch data from Elastic Database periodically using python?

Question

I need to fetch data from Elastic database every 4 minutes, but I am facing problems in how to modify the @timestamp variable in the below mentioned query so as I can push the appropriate query to fetch the data from the URL. I am using Python as the language.

Curl:

curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
            {
             "query": {
               "query_string": {
                  "query": "@timestamp:[2018-06-29T06:47:40.000Z TO *]"
                }
              },
              "size": 1000
            }
            '|json_pp )

I can use CRON to run the script scheduled every 7 minutes, but I can't understand how can I modify the @timestamp variable in the above query so as I can get every new data since the last run.

Any inputs are valuable.

Answer 1

You can use command date in Bash to format timestamp.

current date and time

date +%Y-%m-%dT%H:%M:%S

# 2018-07-14T03:00:58

minus 7 minutes

date --date '-7 min' +%Y-%m-%dT%H:%M:%S

# 2018-07-14T02:53:58

Using `` (ticks/backticks) you can try to put it in other command in Bash (but you many need to use " " instead of ' ' in -d )

curl -XGET "URL" -H 'Content-Type: application/json' -k -u u_name:XXX -d'
            {
             "query": {
               "query_string": {
                  "query": "@timestamp:[`date --date \'-7 min\' +%Y-%m-%dT%H:%M:%S`.000Z TO *]"
                }
              },
              "size": 1000
            }
            '|json_pp )

If you need it as Python code then you can use page https://curl.trillworks.com/ to convert curl to requests and later you can make modifications.

import requests
import datetime
import pprint # pretty print

#dt = datetime.datetime(2018, 6, 29, 6, 47, 40)

dt = datetime.datetime.now()

td_7mins = datetime.timedelta(minutes=7)

dt = dt - td_7mins # now - 7 minutes 

#timestamp = "@timestamp:[{}.000Z TO *]".format(now.strftime("%Y-%m-%dT%H:%M:%S"))
timestamp = dt.strftime("@timestamp:[%Y-%m-%dT%H:%M:%S.000Z TO *]")

data = {
    "query": {
        "query_string": {
            "query": timestamp
        }
      },
    "size": 1000
}
print(data)

url = "https://httpbin.org/get" # good for tests

r = requests.get(url, json=data, headers=headers, verify=False, auth=('u_name', 'XXX'))

pprint.pprint(r.json())