在ElasticSearch上索引聚合数据的最佳方法是什么

Question

I have users and my users have events. Each event as a type and date on each the event happened.

For example

{
  id: 1,
  name: john,
  events: [{
    type: 'logged_in'
    date: "01/01/2016
  },{
    type: 'logged_in'
    date: "02/01/2016
  }{
    type: 'added_email'
    date: "02/05/2016
  }]
}

Now the issue is that I would like to able to query users that have done a specific event N times for a specific time frame

For example: Which users logged in more than twice between Jan 1 16 and Jan 20 17

I know I can use aggregates but the query gets complex and performance drops on million of events.

I was wondering if there is a better way to index/query this data?

Answer 1

The obvious way of representing this data is with a nested mapping:

"id": {"type": "integer"},
"name": {"type": "keyword"},
"events": {
  "type": "nested",
  "properties": {
    "type": {"type": "keyword"},
    "date": {"type": "date"}
  }    

I think this is what you are talking about when you mention performance issues (nested queries and aggregations are slow). For the kind of analysis you're talking about, I don't think you can avoid using an aggregation, but I would "flatten" the data to avoid using nested fields[1], with one document per record instead, like this:

"id": {"type": "integer"},
"name": {"type": "keyword"},
"event_type": {"type": "keyword"},
"date": {"type": "date"}

And then do an aggregation like:

{
  "query": {"bool": {
    "filter": [
      {"match": {"event_type": "logged_in"}},
      {"range": {"date": {"gte": "2016-01-01", "lt": "2017-01-20"}}}
    }
  "aggs": {
    "terms": {
      "field": "name",
      "size": 50
    }
  }

You can also aggregate your data some in your index, in case you know you'll never need more fine-grained analysis. Like for example:

"name": {"type": "keyword"},
"event_type": {"type": "keyword"},
"event_count": {"type": "integer"},
"date_bucket": {"type": "date"}

where date_bucket represents the beginning of the date bucket (like if you only care about full months, then every event for January will go into the record for "2017-01-01"). You can use a scripted updated with upsert to update the event_count in case it already exists, or create a new doc if it doesn't. Then you can use a sum aggregation over event_count inside a terms aggregation instead. This really only makes sense if you know in advance which granularity you care about.

[1] If you also need to access the data in a different way, you might consider indexing into two indices, like two "views" on the data. Basically unless you have infinite resources, or small dataset, or you don't care much about performance, you should work really hard to avoid nested fields.