使用Spark SQL在1.5亿个mongodb集合上提供同步服务的最佳方法是什么？

Question

I have a 150M document MongoDB collection in a single MongoDB instance. Each document is a product. A product has a price and a category. ie:

{
  category: "shoes",
  price: 20,
  .
  .
  .
} 

I would like to expose a rest API method to make synchronous queries over this collection. Ie: what is the average price of all the products for the given category X.

So far, I have tried to implement it in two different ways -- and both seem too slow for exposing a synchronous service (the client would have to wait too long):

1. Using native MongoDB aggregators : Using native MongoDB aggregators seems too slow when the number of products to sum is really big.

2. MongoDB + Spark SQL : Using filtering push down to get the products of a given category and processing the average price within the spark cluster nodes. It takes too long for this approach to load into the cluster memory the products collection. (it took 13 min for a collection of 80k products in an AWS EMR cluster with 1 master and 2 slaves)

So my questions are :

a) Should approach #2 work? Should this approach be fast enough and so I am doing something wrong?

b) What is the best way to achieve this? What is the best solution to achieve this from the architecture point of view?

c) How would you do that?

Many thanks!

Answer 1

Querying a 150M documents collection sitting on a single server expecting high speed seems a too much of an ask in my humble opinion.

With regards to option a), an aggregation pipeline would be executed across all the shards of the collection (unless the $match is on the shard key). Each node would then take care of finding the ones in their own shard, hence distributing the workload. That should provide faster response times (and CPU time for other concurrent queries, if any).

With regards to option b), if I understand correctly, you would end up streaming 150M records through Spark. I am not sure where is the advantage you see coming from this approach.

Therefore, regarding c), the TL;DR is aggregation on sharded collection .