Elasticsearch：获取汇总数据最后条目的摘要

Question

I have an Elasticsearch index with documents like this:

entity_id	operation	timestamp
a1	X	2021-01-01
a1	Y	2021-01-02
a1	Z	2021-01-10
b1	Z	2021-01-03
b1	Z	2021-01-05
b1	Y	2021-01-20
c1	Z	2021-01-03
c1	X	2021-01-05
c1	Y	2021-01-20

There are some entities (entity_id), each of them can be updated multiple times in various ways (operation) in different time (timestamp).

I need cumulative information about last operation executed to each entity. For example, for these data, I need information in form: X=0, Y=2, Z=1

Y=2 because "Y" is the last operation happened to "b1" and "c1" entities

Z=1 because "Z" is the last operation happened to "a1" entity

I made the query to get info on last operation for each entity, like below:

{
    "size": 0,
    "aggs": {
        "group_by_id": {
            "terms": {
                "field": "entity_id"
            },
            "aggs": {
                "last_entry": {
                    "top_hits": {
                        "size": 1,
                        "_source": {
                            "include": [
                                "operation",
                                "timestamp"
                            ]
                        },
                        "sort": [
                            {
                                "timestamp": {
                                    "order": "desc"
                                }
                            }
                        ]
                    }
                }
            }
        }
    }
}

It works but due to huge amount of data I won't be able to iterate on the aggregation results and sum operations by type afterwards, in code. I need to count of last operations in Elasticsearch query, if this feasible.

Can anybody suggest how this can be achieved?

Thanks!

Answer 1

I have been searching for a while for approach to solve tasks like this. I have found lots of similar questions but no proper suggestions so far. Here is the solution I has finally come to, maybe it will be useful for someone with similar tasks. The idea is to use scripted_metric aggregation and calculate needed summary data by means of scripts

{
"size": 0,
"aggs":{
    "total": {
        "scripted_metric": {
            "init_script": "state.operations=new Hashtable();", 
            "map_script": <Add to state.operations every doc using entity_id as key. When another doc for the same entity_id is found check its timestamp and replace the existing doc if the new found doc is newer>,
            "combine_script": "return state.operations",
            "reduce_script": <Here you have "states" variable which contains hashtables returned by the combine script per each shard. You can iterate states, merge all hashtables together and return the resulting hashtable or just calculate needed summary values>
        }
    }
},
"sort": [
    {
        "timestamp": {
            "order": "desc"
        }
    }
]

}

This is just an algorithm, I wrote plain description in map_script and combine_script as my real case was much more complex than the simplified example I posted here.