批量导入数据到ElasticSearch

Question

I have elasticsearch data in JSON that i wanna upload at once via curl

curl -s -H "Content-Type: application/json" -XPOST localhost:9200/_bulk --data-binary @C:\Users\adm\Desktop\test.json

but I get this error:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_score]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_score]"},"status":400}

and the data (test.json) looks like this:

{"index" :{"_index":"variationdetails","_type":"_doc","_id":"e17bd50b-fe65-423c-a9f8-4d45ecf56559","_score":1,"_source":{"entityname":"Cislo_f","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"daňový doklad č.","keypositionbottom":301,"keypositionleft":1482,"keypositionright":2000,"keypositiontop":251,"keytovaluedeltaleft":551,"keytovaluedeltatop":7,"userchanged":true,"valuepositionbottom":306,"valuepositionleft":2033,"valuepositionright":2387,"valuepositiontop":258,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}}
{"index" :{"_index":"variationdetails","_type":"_doc","_id":"c2a831f1-8156-434c-bd84-08db64c935a5","_score":1,"_source":{"entityname":"Datum_splatnosti","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"Datum splatnosti:","keypositionbottom":1154,"keypositionleft":1706,"keypositionright":2015,"keypositiontop":1112,"keytovaluedeltaleft":421,"keytovaluedeltatop":11,"userchanged":true,"valuepositionbottom":1149,"valuepositionleft":2127,"valuepositionright":2298,"valuepositiontop":1123,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"c2a831f1-8156-434c-bd84-08db64c935a5"}}}

I tried changing _bulk to variationdetails/_doc but that didnt help. I cant use elasticdump on the target system (no inte.net or copy files option)

Answer 1

The documentation for the bulk insert API gives an example and description of the required input.

For each record you want to create or update, you need two lines of JSON:

• The first line specifies the action to take , and the document to take it on . Essentially, the details which would be in the URL and HTTP request method on a single-item action.
• The second line specifies the data to use. Essentially, the details which would be in the body of a single-item action.

So for your example, it would look like this:

{"index" :{"_index":"variationdetails","_id":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}
{"_type":"_doc","_score":1,"_source":{"entityname":"Cislo_f","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"daňový doklad č.","keypositionbottom":301,"keypositionleft":1482,"keypositionright":2000,"keypositiontop":251,"keytovaluedeltaleft":551,"keytovaluedeltatop":7,"userchanged":true,"valuepositionbottom":306,"valuepositionleft":2033,"valuepositionright":2387,"valuepositiontop":258,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}
{"index" :{"_index":"variationdetails","_id":"c2a831f1-8156-434c-bd84-08db64c935a5"}}
{"_type":"_doc","_score":1,"_source":{"entityname":"Datum_splatnosti","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"Datum splatnosti:","keypositionbottom":1154,"keypositionleft":1706,"keypositionright":2015,"keypositiontop":1112,"keytovaluedeltaleft":421,"keytovaluedeltatop":11,"userchanged":true,"valuepositionbottom":1149,"valuepositionleft":2127,"valuepositionright":2298,"valuepositiontop":1123,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"c2a831f1-8156-434c-bd84-08db64c935a5"}}

I'm not sure if _source is supposed to be part of the document or not; if not, you probably want this:

{"index" :{"_index":"variationdetails","_id":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}
{"_type":"_doc","_score":1,"entityname":"Cislo_f","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"daňový doklad č.","keypositionbottom":301,"keypositionleft":1482,"keypositionright":2000,"keypositiontop":251,"keytovaluedeltaleft":551,"keytovaluedeltatop":7,"userchanged":true,"valuepositionbottom":306,"valuepositionleft":2033,"valuepositionright":2387,"valuepositiontop":258,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}
{"index" :{"_index":"variationdetails","_id":"c2a831f1-8156-434c-bd84-08db64c935a5"}}
{"_type":"_doc","_score":1,"entityname":"Datum_splatnosti","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"Datum splatnosti:","keypositionbottom":1154,"keypositionleft":1706,"keypositionright":2015,"keypositiontop":1112,"keytovaluedeltaleft":421,"keytovaluedeltatop":11,"userchanged":true,"valuepositionbottom":1149,"valuepositionleft":2127,"valuepositionright":2298,"valuepositiontop":1123,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"c2a831f1-8156-434c-bd84-08db64c935a5"}

Answer 2

If you want to transfer data from cluster to another then best option is to use Snapshot and Restore API of elasticsearch.

If you want to use _bulk API then you need to follow bulk api format and your json format should be in below format only. You can create your json file in ndjson format for bulk api.

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n

{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }

You are getting error for _score because it is internal field / variable of elasticsearch and it show value of relevancy score based on your query.