将数据放入Elasticsearch时，如何处理有时具有不同结构的字段？

Question

I'm getting several mapping Exceptions when trying to insert data from my mongoDb into Elastic. After some investigative work, it seems that the error comes from the fact that I have a field in my db that is sometimes and array of strings, while other times an array of objects.

Meaning, for some documents in mongo it will have this:

{"my_field" : ["one", "two"]

while others

{"my_field": [{"key":"value", "key2":"value"}, {"key":"value", "key2":"value"}, ...]

I'm having a difficult time in pinning down how exactly this situation is handled in Elastic.

Answer 1

You will need to massage the data before it is indexed so that it does conform to elasticsearch's rules. One approach is for my_field to be a nested document - for one document you might have

{"my_field": {"string_value": ["one", "two"]}}

and for another

{"my_field": {"doc_value": {"key":"value", "key2":"value"}}}

This assumes that the values for key and key2 will always have the same type and that there is a small number of possible keys in this document. If this document contains arbitrary data you might be better off indexing as

{"my_field": [{"key": "key1", "string_value": "value"},
            {"key": "key2", "int_value": "123"}]}

As for how you massage, one option is to do this before you send the data to elasticsearch. The downside is that the the _source attribute will obviously contained the transformed data.

Another approach is to send the data to elasticsearch as-is, but to have a transform in the mapping that elasticsearch will run to transform the data before indexing.