如何快速聚合大量数据

Question

I need to aggregate all the keywords in the news for a period of time, for example:

{
  "news_ID":"123456",
  "news_content":"Apple pencil",
  "keywords": {
      [
        {
         "word" : "Apple",
         "score" : 0.0653220043
        },
        {
         "word" : "pencil",
         "score" : 0.7096893191
        }
      ]
    },
  "publish_time":"2020-01-03"
}

I want to know how many times apple appeared between 2020-01 and 2020-02, but there are too many keywords...

Could you please advise me on how I should approach this requirement as per best practices?

Answer 1

Syncing a sample doc:

PUT tester/_doc/1
{
  "news_ID":"123456",
  "news_content":"Apple pencil",
  "keywords":[
    "apple",
    "pencil"
  ],
  "publish_time":"2020-01-03"
}

Using a terms aggregation w/ a range filter on the top level:

GET tester/_search
{
  "size": 0,
  "query": {
    "range": {
      "publish_time": {
        "gte": "2020-01-01",
        "lt": "2020-02-01"
      }
    }
  },
  "aggs": {
    "by_keywords": {
      "terms": {
        "field": "keywords.keyword"
      }
    }
  }
}

You can also use a filtered aggregation to aggregate on multiple monthly buckets:

GET tester/_search
{
  "size": 0,
  "aggs": {
    "2020-01_2020-02": {
      "filter": {
        "range": {
          "publish_time": {
            "gte": "2020-01-01",
            "lt": "2020-02-01"
          }
        }
      },
      "aggs": {
        "by_keywords": {
          "terms": {
            "field": "keywords.keyword"
          }
        }
      }
    },
    "2020-02_2020-03": {
      "filter": {
        "range": {
          "publish_time": {
            "gte": "2020-02-01",
            "lt": "2020-03-01"
          }
        }
      },
      "aggs": {
        "by_keywords": {
          "terms": {
            "field": "keywords.keyword"
          }
        }
      }
    }
  }
}