Elasticsearch：搜索类似产品

Question

I have a list of 50 million products. Each product has a list of 200 features. I am looking to find similar products by matching product features which has maximum overlap across the 200 features.

Currently I concatenate the 200 words with spaces and form a long string. When I want to find similar products for a particular selected product, I retrieve the stored 200 words long string and search elasticsearch.

This gives expected results, but each search takes roughly around 7 seconds. That is because the search phrase is so long. Is there a better way to do this and find best overlap on elastic ?

Answer 1

I suggest that you check/try few things:

I have a list of 50 million products. Each product has a list of 200 features. I am looking to find similar products by matching product features which has maximum overlap across the 200 features. Currently I concatenate the 200 words with spaces and form a long string.

Assuming Product is a Doc type you could give it a try save features normally as array of values and enabling field data on it. It would then be easy to use aggregations to group them applying the proper mentioned max overlap and get what you want. I strongly believe it would be much faster.

I retrieve the stored 200 words long string and search elasticsearch.

Might have some situation where all you want is the aggregation result and not the full response with all product or other doc type, in this cases set search type to count(older versions) or query_then_fetch with size 0(newer versions) and you avoid an initial fetch of all doc types and get only the aggregations, this could be used in some situations based on your requirements.

Make sure you have elasticsearch environment proper prepared .

Finally with this number of docs there's a chance that you find some shard /replicas configuration that suits better your case than the default one.