improving performance with elastic search and python

Question

I am reading data from my elastic cluster and then converting it into a pandas data frame. I want to do some analysis on the data frame and then again i will visualizing it. I want to make it real time. But i am getting very slow response from my elastic cluseter and most of the times i am gettign the below error :-

elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=10))

My code to perform the above operation is:-

import pandas as pd
import datetime
import elasticsearch
import elasticsearch.helpers
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search
from pandasticsearch import DataFrame
from pandasticsearch import Select
from elasticsearch import Elasticsearch, helpers
import os


# Define the client which will be our elastic cluster URL
client = Elasticsearch(['http://localhost:9200/'])

# Define search method on the client by using the Search function.
# make sure that the Search function start with Capital S (Search(using=client)) as this is a function.
search = Search(using=client)

# Get all the results from the search method and store it in result to know how many hits we are getting.
results = search.execute()

# To know about the total number of hits we are getting run the below chunk.
results.hits.total  # 2351472834 (I got 2.3 billion hits as a result)

# Again I am defining a method s on which we will perform the query. you have to run this method everytime before running the query.
s = Search(using=client)

# add any filters/queries....

# The below line you can use if you want to dump all the data and in this case we have 2.3 billion observation.
#s = s.query({"match_all": {}})

# In the below code you can add filters,queries or time constraints.
s = s.query({"constant_score": {
            "filter": {
                "bool": {
                    "must": [{
                        "range": {"@timestamp": {
                             "gte": "2019-05-15T14:00:00.000Z",  # gte - greater than
                             "lte": "2019-05-15T14:30:00.000Z"  # lte - less than

                             }}
                    }],
                    "filter": [
                        # 1st filter, get all the data where type is "vx_apache_json"
                        {"term": {"type": "vx_pp_log"}},
                        # 2nd filter, get all the data where domain is "fnwp"
                        {"term": {"domain": "fnwp"}},
                        # 3rd filter, get all the data where RTP:a is "end"
                        {"term": {"RTP:a": "end"}},
                    ]

                }}}})

# After getting all the result in the variable s, we are applying scan method on it and converting it into a data frame.
results_df = pd.DataFrame((d.to_dict() for d in s.scan()))

# TO have a look at the data frame use the below name of the data frame
# results_df

results_df.to_csv('signin.csv', index=False)

I am just reading 30 minutes of data and i want to do it on may be on 24 hours or 4 hours depending on my need which is below in my filter :-

"gte": "2019-05-15T14:00:00.000Z",  # gte - greater than
"lte": "2019-05-15T14:30:00.000Z"  # lte - less than

Answer 1

Since it's hard to optimize a search query without access to the elasticsearch, i can only tell you how to treat the ReadTimeout Error, by raising the timeout.

client = Elasticsearch([' http://localhost:9200/ '],timeout=60, max_retries=10, retry_on_timeout=True)