如何在非常大的数据库上使用mongodb查询操作（每个分片有3个大约260-3亿）

Question

I have to find data in between different date ranges column in a sharded database having total of around 800 million documents. I am using this query:

cursordata=event.aggregate([{"$match":{}},{"$unwind":},{"$project":{}}])

However, when I change it to a pandas dataframe

df=pd.DataFrame(cursordata)

It is taking for ever and not working at all, it just got stuck.

I have 2 choices:

1. Either keep doing query for different conditions directly from mongodb or
2. After changing to data to dataframe, perform operation for different conditions

Please suggest how to proceed.

Answer 1

Could we have a sample of documents? I think you should look for an index matching the fields you're querying.

As a reminder, try to keep in mind the Equality, Sort, Range rule in MongoDB indexing.
Besides, since you're in a sharded cluster you might want to have your sharding key in you query, otherwise the mongos will query all the shards (more info here )