在Elastic Search中，如何一次批量索引Json文件多值文档？

Question

I've already looked through ES documentation and read related questions, but none of them have worked for me thus far.

Basically I have a Json file with written with multiple documents in this format:

[ { 
    "account": "Sam420", 
    "language": null, 
    "watchers": 0, 
    "commits": 14, 
    "contributors": 2, 
    "stars": 0, 
    "rank": 16, 
}
{ 
    "account": "Kelly", 
    "language": null, 
    "watchers": 0, 
    "commits": 14, 
    "contributors": 2, 
    "stars": 0, 
    "rank": 16, 
} ] 

I have tried post request using the bulk API to my local ES setup with this following the following body format:

 { "index": {} }
 { 
    "account": "Kelly", 
    "language": null, 
    "watchers": 0, 
    "commits": 14, 
    "contributors": 2, 
    "stars": 0, 
    "rank": 16, 
} 
{ "index": {} }
{ 
    "account": "Kelly", 
    "language": null, 
    "watchers": 0, 
    "commits": 14, 
    "contributors": 2, 
    "stars": 0, 
    "rank": 16, 
} 

But, I'm getting a parser error. It does work when I rearrange data into one single row per data like this:

{ "index": { "_index": "folder" } }
{ "account": "Sam420", "language": null, ... }
{ "index": { "_index": "Canigan"} }
{ "account": "Kelly", "language": null, ... } 

Here's the parser error:

{
    "error": {
    "root_cause": [
       {
          "type": "json_parse_exception",
          "reason": "Unexpected character (':' (code 58)): expected a      
                    valid value (number, String, array, object, 'true',     
                    'false' or 'null')\n at [Source: [B@6bd0ddf7; line: 
                    1, column: 10]"
        }],
           "type": "json_parse_exception",
           "reason": "Unexpected character (':' (code 58)): expected a 
                     valid value (number, String, array, object 'true', 
                    'false' or 'null')\n at [Source: [B@6bd0ddf7; line: 
                    1, column: 10]"
       },
       "status": 500
}

But, I'm pulling repo data with 100+ documents from Github API, and each value is arranged vertically. Without having to reformat it using script, what can I do to bulk index multiple documents in the Json format that's already given to me? If not, is there any other way besides bulk index I can use to index multiple documents at once?

Answer 1

The documentation for version 5.5 makes this pretty clear: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html

Because this format uses literal \\n's as delimiters, please be sure that the JSON actions and sources are not pretty printed.

You must lay out the objects as single lines.

That being said, you don't really need a complicated script to reformat your objects. You could use something like Notepad++ to replace ",\\n" (comma then newline) with ", " (comma then space). Then interleave your index/metadata lines like you're doing.

You might also need to watch out for the trailing comma at the end of your list of properties.

Answer 2

I think it doesn't work because you have no information about the index and the index type.

{ "index": {"my_index", "my_index_type"} }
{ "account": "Sam420", "language": null, ... }
{ "index": {"my_index", "my_index_type"} }
{ "account": "Kelly", "language": null, ... }