如何通过 Elasticsearch 6.x 中的动态或未知字段进行聚合

Question

I'm fairly new to ElasticSearch, currently using v6.2 and I seem to have run into a problem while trying to add some aggregations to a query. Trying to wrap my head around the various types of aggregation, as well as the best ways to store the data.

When the query runs, I have some variable attributes that I would like to aggregate and then return as filters to the user. For example, one character may have attributes for "size", "shape" and "colour", while another only has "shape" and "colour".

The full list of attributes is unknown so I don't think I would be able to construct the query that way.

My data is currently structured like this:

{
    id : 1,
    title : 'New Character 1',
    group : 1,
    region : 1,
    attrs : [
        moves : 2,

        # These would be dynamic, would only apply to some rows, not others.
        var_colours : ['Blue', Green', 'Red'],
        var_shapes : ['Round', 'Square', 'Etc'],

        effects : [
            { id : 1, value: 20},
            { id : 2, value: 60},
            { id : 3, value: 10},
        ]

    ]
}

I currently have an aggregation of groups and regions that looks like this. It seems to be working wonderfully and I would like to add something similar for the attributes.

[
    'aggs' => [
        'group_ids' => [
            'terms' => [
                'field' => 'group',
                'order' => [ '_count' => 'desc' ]
            ]
        ],
        'region_ids' => [
            'terms' => [
                'field' => 'region',
                'order' => [ '_count' => 'desc' ]
            ]
        ]
    ]
]

I'm hoping to get a result that looks like the below. I am also not sure if the data structure is setup in the best way either, I can make changes there if necessary.

[aggregations] => [
    [groups] => [
        [doc_count_error_upper_bound] => 0
        [sum_other_doc_count] => 0
        [buckets] => [
            [0] => [
                [key] => 5
                [doc_count] => 27
            ],
            [1] => [
                [key] => 2
                [doc_count] => 7
            ]
        ]
    ],

    [var_colours] => [
        [doc_count_error_upper_bound] => 0
        [sum_other_doc_count] => 0
        [buckets] => [
            [0] => [
                [key] => 'Red'
                [doc_count] => 27
            ],
            [1] => [
                [key] => 'Blue'
                [doc_count] => 7
            ]
        ]
    ],

    [var_shapes] => [
        [doc_count_error_upper_bound] => 0
        [sum_other_doc_count] => 0
        [buckets] => [
            [0] => [
                [key] => 'Round'
                [doc_count] => 27
            ],
            [1] => [
                [key] => 'Polygon'
                [doc_count] => 7
            ]
        ]
    ]

    // ...
]

Any insight that anyone could provide would be extremely appreciated.

Answer 1

You should do this within your PHP script.

I can think of the following:

1. Use the Dynamic field mapping for your index.

By default, when a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping. This behaviour can be disabled, both at the document and at the object level, by setting the dynamic parameter to false (to ignore new fields) or to strict (to throw an exception if an unknown field is encountered).

2. Get all the existing fields in your index. Use the Get mapping API for this.

3. Loop over the results of Step 2 so you can get all the existing fields in your index. You can store them in a list (or array), for example.

4. You can create a PHP Elasticsearch terms aggregation for each of the fields in your list (or array). This is: create an empty or base query with no terms aggregation and add one terms for each element you got from step 3.

5. Add to each terms, the missing field with an empty empty string ("").

6. That's it. Following this, you have creating a query in such way that, no matter what index you're searching, you'll get a terms agg with all the existing fields for it.

Advantages :

• Your terms aggregations will be generated dynamically with all the existing fields.

• For each of the doc that does not contain any of the fields, an empty string will be shown.

Disadvantages :

• Looping through the GET mapping API's result could be a little frustrating (but I trust you).

• Performance (time & resources) will be affected for every new field you find in your mappings.

I hope this is helpful: :D