Elasticsearch Terms Query exclude large amount of users

Question

I'm working on a tinder like app. In order to exclude profiles that user has swiped before, I use a "must_not" query like this:

must_not : [{"terms": { "swipedusers": ["userid1", "userid1", "userid1"…]}}]

I wonder what are the limits using this approach? is this a scalable approach that would also work when the swipedusers array contains 2000 user ids? If there is a better scalable approach to this I would be happy to know...

Answer 1

there is a better approach! and it called "terms lookup", is something like the traditional join that you could do on relational databases...

I could try to explain you here, but, all the information that you need is well documented on the official Elastic Search page:

https://www.elastic.co/guide/en/elasticsearch/reference/5.0/query-dsl-terms-query.html#query-dsl-terms-lookup

The final solution is having 2 indices, one for the registered users and another one to track swipes for each user. Then, for each swipe, you should update the document containing current user swipes... Here you will need to add elements to an array, and this is another problem in ElasticSearch (big problem if you are using AWS managed ElasticSearch) that only can be solved using scripting... More info at https://www.elastic.co/guide/en/elasticsearch/guide/current/partial-updates.html#_using_scripts_to_make_partial_updates

For your case, the query will result in something like:

GET /possible_matches/_search
{
    "query" : {
        "terms" : {
            "user" : {
                "index" : "swiped",
                "type" : "users",
                "id" : "current-user-id",
                "path" : "swipedUserId"
            }
        }
    }
}

Another thing that you should take in account is the replication configuration for the swipes index, since each node will perform "joins" with that index, is highly recommended to have a full copy of that index in each node. You could achieve this creating the index with the "auto_expand_replicas" with "0-all" value.

PUT /swipes
{
    "settings": {
        "auto_expand_replicas": "0-all"
    }
}