如何从650 M数据中获取Elasticsearch中特定字段值的出现次数

Question

I have indexed Twitter data in ES. There are 110 M Twitter unique users profiles and there 650 M Tweets . Both are in seperate index (index: twitter-profiles, type: profiles), for tweets (index: twitter-tweets, type: tweets).

There is user_id_str of profile is attached with every tweet.

I am running into a problem to get occurrence count of specific user . I used Facet/terms and Aggregation/Terms but both give me exception PartialShardFailureException because there are lot of data to make calculation. I used following query

{
"aggs" : {
    "userCount" : {
        "terms" : { "field" : "user_id_str" }
    }
  }
}

Then I give another Try.

I used second method Scan . Here I get ids of profiles from profiles type then search it in tweet type. it give me results but a single result came after 2seconds OOps. There are 110 M users mean I have to wait for days.

Please give me any reasonable solution for this situation.

Answer 1

您可以结合使用基数聚合和术语过滤器