批量導入數據到ElasticSearch

Question

我在 JSON 中有 elasticsearch 數據，我想通過 curl 立即上傳

curl -s -H "Content-Type: application/json" -XPOST localhost:9200/_bulk --data-binary @C:\Users\adm\Desktop\test.json

但我收到此錯誤：

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_score]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_score]"},"status":400}

數據 (test.json) 如下所示：

{"index" :{"_index":"variationdetails","_type":"_doc","_id":"e17bd50b-fe65-423c-a9f8-4d45ecf56559","_score":1,"_source":{"entityname":"Cislo_f","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"daňový doklad č.","keypositionbottom":301,"keypositionleft":1482,"keypositionright":2000,"keypositiontop":251,"keytovaluedeltaleft":551,"keytovaluedeltatop":7,"userchanged":true,"valuepositionbottom":306,"valuepositionleft":2033,"valuepositionright":2387,"valuepositiontop":258,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}}
{"index" :{"_index":"variationdetails","_type":"_doc","_id":"c2a831f1-8156-434c-bd84-08db64c935a5","_score":1,"_source":{"entityname":"Datum_splatnosti","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"Datum splatnosti:","keypositionbottom":1154,"keypositionleft":1706,"keypositionright":2015,"keypositiontop":1112,"keytovaluedeltaleft":421,"keytovaluedeltatop":11,"userchanged":true,"valuepositionbottom":1149,"valuepositionleft":2127,"valuepositionright":2298,"valuepositiontop":1123,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"c2a831f1-8156-434c-bd84-08db64c935a5"}}}

我嘗試將 _bulk 更改為 variationdetails/_doc 但這沒有幫助。 我不能在目標系統上使用 elasticdump（沒有 inte.net 或復制文件選項）

Answer 1

批量插入 API 的文檔給出了所需輸入的示例和描述。

對於您要創建或更新的每條記錄，您需要兩行 JSON：

• 第一行指定要采取的行動，以及采取行動的文件。 本質上，在單項操作的 URL 和 HTTP 請求方法中的詳細信息。
• 第二行指定要使用的數據。 本質上，細節將包含在單項操作的主體中。

因此，對於您的示例，它看起來像這樣：

{"index" :{"_index":"variationdetails","_id":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}
{"_type":"_doc","_score":1,"_source":{"entityname":"Cislo_f","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"daňový doklad č.","keypositionbottom":301,"keypositionleft":1482,"keypositionright":2000,"keypositiontop":251,"keytovaluedeltaleft":551,"keytovaluedeltatop":7,"userchanged":true,"valuepositionbottom":306,"valuepositionleft":2033,"valuepositionright":2387,"valuepositiontop":258,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}
{"index" :{"_index":"variationdetails","_id":"c2a831f1-8156-434c-bd84-08db64c935a5"}}
{"_type":"_doc","_score":1,"_source":{"entityname":"Datum_splatnosti","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"Datum splatnosti:","keypositionbottom":1154,"keypositionleft":1706,"keypositionright":2015,"keypositiontop":1112,"keytovaluedeltaleft":421,"keytovaluedeltatop":11,"userchanged":true,"valuepositionbottom":1149,"valuepositionleft":2127,"valuepositionright":2298,"valuepositiontop":1123,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"c2a831f1-8156-434c-bd84-08db64c935a5"}}

我不確定_source是否應該成為文檔的一部分； 如果沒有，你可能想要這個：

{"index" :{"_index":"variationdetails","_id":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}}
{"_type":"_doc","_score":1,"entityname":"Cislo_f","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"daňový doklad č.","keypositionbottom":301,"keypositionleft":1482,"keypositionright":2000,"keypositiontop":251,"keytovaluedeltaleft":551,"keytovaluedeltatop":7,"userchanged":true,"valuepositionbottom":306,"valuepositionleft":2033,"valuepositionright":2387,"valuepositiontop":258,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"e17bd50b-fe65-423c-a9f8-4d45ecf56559"}
{"index" :{"_index":"variationdetails","_id":"c2a831f1-8156-434c-bd84-08db64c935a5"}}
{"_type":"_doc","_score":1,"entityname":"Datum_splatnosti","keyFieldObject":null,"keynotcolumn":false,"keyphrase":"Datum splatnosti:","keypositionbottom":1154,"keypositionleft":1706,"keypositionright":2015,"keypositiontop":1112,"keytovaluedeltaleft":421,"keytovaluedeltatop":11,"userchanged":true,"valuepositionbottom":1149,"valuepositionleft":2127,"valuepositionright":2298,"valuepositiontop":1123,"variationguid":"a20e3d7a-bf38-4eae-9f23-fb100b539d08","vddid":"c2a831f1-8156-434c-bd84-08db64c935a5"}

Answer 2

如果您想將數據從集群傳輸到另一個集群，那么最好的選擇是使用 elasticsearch 的快照和還原API。

如果你想使用_bulk API那么你需要遵循 bulk api 格式，你的 json 格式只能是下面的格式。 您可以為批量 api 創建ndjson格式的 json 文件。

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n

{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }

您收到_score的內部字段/變量，並且它根據您的查詢顯示相關性分數的值。