使用 Python 客户端通过映射将不规则 json 加载到 Elasticsearch 索引中

Question

I have some .json where not all fields are present in all records, for eg caseclass.json looks like:

[{
    "name" : "john smith", 
    "age" : 12, 
    "cars": ["ford", "toyota"], 
    "comment": "i am happy"
},
{
    "name": "a. n. other", 
    "cars": "", 
    "comment": "i am panicking"
}]

Using Elasticsearch-7.6.1 via python client elasticsearch:

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search
import json
import os
from elasticsearch_dsl import Document, Text, Date, Integer, analyzer

es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
class Person(Document):
        class Index:
            using = es
            name = 'person_index'
        name = Text()
        age = Integer()
        cars = Text()
        comment = Text(analyzer='snowball')   

Person.init()

with open ("caseclass.json") as json_file:
data = json.load(json_file)
for indexid in range(len(data)):
    document = Person(name=data[indexid]['name'], age=data[indexid]['age'], cars=data[indexid]['cars'], comment=data[indexid]['comment'])
    document.meta.id = indexid
    document.save()

Naturally I get KeyError: 'age' when the second record is trying to be read. My question is: it is possible to load such records onto a Elasticsearch index using the Python client and a pre-defined mapping , instead of dynamic mapping? Above code works if all fields are present in all records but is there a way to do this without checking presence of each field per record as the actual records have complex structure and there are millions of them? Thanks

Answer 1

The error has nothing to do w/ your mapping -- it's just telling you that age could not be accessed in one of your caseclasses .

The index mapping is created when you call Person.init() -- you can verify that by calling print(es.indices.get_mapping(Person.Index.name)) right after Person.init() .

I've cleaned up your code a bit:

import json
import os
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search, Document, Text, Date, Integer, analyzer

es = Elasticsearch([{'host': 'localhost', 'port': 9200}])


class Person(Document):
    class Index:
        using = es
        name = 'person_index'
    name = Text()
    age = Integer()
    cars = Text()
    comment = Text(analyzer='snowball')


Person.init()
print(es.indices.get_mapping(Person.Index.name))

with open("caseclass.json") as json_file:
    data = json.load(json_file)
    for indexid, case in enumerate(data):
        document = Person(**case)
        document.meta.id = indexid
        document.save()

Notice how I used **case to spread all key-value pairs inside of a case instead of using data[property_key] .

The generated mapping is as follows:

{
  "person_index" : {
    "mappings" : {
      "properties" : {
        "age" : {
          "type" : "integer"
        },
        "cars" : {
          "type" : "text"
        },
        "comment" : {
          "type" : "text",
          "analyzer" : "snowball"
        },
        "name" : {
          "type" : "text"
        }
      }
    }
  }
}