Import data into Elastic Search using Logstash in Python

Question

I have imported a JSON file into ElasticSearch index using the command line for logstash. I used the following command:

.\\logstash -f logstash.conf

The path of the file to be imported has been provided in the logstash.conf file. I have been able to achieve this using command line.

But how do i do the same thing in python? Aim is to import bulk data into elasticsearch index using python (with or without logstash)

Answer 1

You can use bulk loader http://elasticsearch-py.readthedocs.io/en/master/helpers.html?highlight=bulk#bulk-helpers

in case you're not able to fit your file in memory - use streaming option http://elasticsearch-py.readthedocs.io/en/master/helpers.html?highlight=bulk#elasticsearch.helpers.streaming_bulk