When indexing data using bulk API of elasticsearch here is the sample json from the site documentation
POST _bulk
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "2" } }
{ "field1" : "value2" }
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "3" } }
{ "field1" : "value3" }
While "preparing" the data to be used by the bulk API, on first line I have to specify the operation and in next line I will provide data. Some redundant parts on each line might look obvious and pretty harmless but when I am indexing trillions of rows, doesn't it add up to latency? Is there is better way to push all the rows by specifying the index name and type only once at the header? Specially when I can use autogenerated id, I can avoid generating terabytes of data just to be prepended to every row for the same purpose again and again.
I believe I am missing something obvious here otherwise I am sure those guys at elastic are smart enough to have figured it out already and if they have done it this way, there should be some reason. But what?
Here you have shortcut:
POST /test/_doc/_bulk
{ "index": {} }
{ "field1" : "value1" }
{ "index": {} }
{ "field1" : "value2" }
{ "index": {} }
{ "field1" : "value3" }
Unfortunately you still need to repeat the { "index": {} }
line but index name and document type you have specified in the path.
Please see more options in the Cheaper in Bulk article.
Well there's no better way in terms of preparing the data. Only thing you can do is to prepare the data programmatically.
You can simply write a code to construct the desired json and send it across using _bulk
API.
Ideally it is best to have indexing done via a specific application called indexer
which would actually wait until a batch of documents for eg 50
or 100
is collected and then execute the _bulk
API programmatically.
Or instead of batch processing like that, you can have it document by document ie event based
using messaging queues. (Best approach to minimize latency during indexing process)
Another option is to create an input file say data.json
(purely batch processing) using a simple java program or any other programming language you use, append all documents you want programmatically and use CURL command to send the request as shown below:
$ curl -s -XPOST <host_name>:9200/_bulk --data-binary @data.json
So for this indexer
application, you can add scheduling as well as mail notifications in such a way that you'd get to know the status of every job run and schedule time as when to run everyday/week depending on your requirement.
Otoh, you can make use of Logstash
. Sorry, its not the best answer, but I hope it helps.
As was already told in Piotr Pradzynski's great answer , there's not much you can do, and the minimal footprint is the one Pyotr proposed. There are a couple of details that I believe deserve to be added.
The main reason to consider bulk API is tuning for indexing speed . The improvements in performance here are largely due to saving on handling less HTTP connections on the Elasticsearch side. Practically speaking, your cluster will not be indexing the documents faster if you manage not to send those repetitive { "index": {} }
parts.
In this case I believe the best one can do is to send the data compressed, like this:
curl -v 'http://localhost:9200/my_index/doc/_bulk' \
-H "Content-encoding: gzip"
-H "content-type: application/json; charset=UTF-8"
-X POST --data-binary @bulk_data.json.gz
To illustrate the idea I generated a file with random data that looks like this:
$ head bulk_data.json
{"index":{}}
{"request_id":"40485"}
{"index":{}}
{"request_id":"12417"}
{"index":{}}
{"request_id":"11945"}
{"index":{}}
{"request_id":"81722"}
{"index":{}}
{"request_id":"52613"}
The size of the file is 10 times smaller after compression with GZip:
$ ls -l
-rw-r--r-- 1 vasiliev staff 358836 Nov 16 20:09 bulk_data.json
-rw-r--r-- 1 vasiliev staff 35744 Nov 16 19:41 bulk_data.json.gz
This might help a lot in the case of limited bandwidth.
Compression is also available from client libraries, like elasticsearch-py library.
Hope that helps!
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.