Use python to get data table Visualization from elastic search

Question

How can I filter and summarize data in elastic search through python. I manually created a data table visualization through Kibana interface and downloaded it in .csv format. Now I want to do the same using python.

For example, if there are 10 variables in the index: v1,v2,v3,.. v10 then how to get a data table which can be described in sql as:

select v2, count(v2) 
from index 
where v1 = "some value" 
group by v2 

Till now I am able to do this:

from elasticsearch5 import Elasticsearch
user = 'xxx'
password = 'xxx'
url = 'xxx'
command = "%s:%s@%s:9200" % (user,password,url)
x = Elasticsearch(command)
# Get the count of documents
num = x.count(index='my_index')['count']
# Get documents filtered by v1
my_docs = x.search(index="my_index",  body={"query": {"match": {'v1':'US'}}})

Now what I want is to select only variable v2 from my_docs and also group by v2 to get a count. Apologies that I don't know how to create a reproducible example without revealing the user credentials.

Answer 1

• First: I do not want to download complete documents (each document in the actual data contain 150+ variables).

If you want to treat only few fields on your doc, you should use the _source filter before your query - doc here . For example to retrieve from your docs only the v1 and v2 fields :

body={
    "_source": ["v1", "v2"],"query": {"match": {'v1':'US'}}}

• Second: I am not familiar with json yet, though I am working on it.

You just try something like this:

for result in mydocs['hits']['hits']:
    print result["_source"]['v1']
    print result["_source"]['v2']