当有数百万条记录时，Mongo计数真的很慢

Question

//FAST
db.datasources.find().count()
12036788

//SLOW    
db.datasources.find({nid:19882}).count()
10161684

Index on nid

Any way to make the second query faster? (It is taking about 8 seconds)

Answer 1

Count queries, indexed or otherwise, are slow due to the fact that MongoDB still has to do a full b-tree walk to find the appropriate number of documents that match your criteria. The reason for this is that the MongoDB b-tree structure is not "counted" meaning each node does not store information about the amount of elements in the node/subtree.

The issue is reported here https://jira.mongodb.org/browse/SERVER-1752 and there is currently no workaround to improve performance other than manually maintaining a counter for that collection which obviously comes with a few downsides.

Also note that the db.col.count() version (so no criteria) can take a big shortcut and doesn't actually perform a query hence it's speed. That said it does not always report the same value as a count query would that should return all elements (it won't be in sharded environments with high write throughput for example). Up for debate whether or not that's a bug. I think it is.

Note that in 2.3+ a significant optimization was introduced that should (and does) improve performance of counts on indexed fields. See : https://jira.mongodb.org/browse/SERVER-7745

Answer 2

As @Remon said, count() has to scan all the documents matching the query/filter. It is O(n) where n is the number of documents that will match the index, or the number of documents in the collection if the field is not indexed.

In such cases, you typically want to revisit your requirement. Do you really need a precise number for the result 10161684? If the precision is important, you should keep a separate counter for the particular query.

But in most cases, precision is not important. It's one of the two:

• You don't care whether it's 10 million or 10.2 million, but the order of magnitude is important, ie, you care about whether it's 8 million or 10 million.
• You only care about the precise number if it's a small one. Ie, you're interested to know that there are 44 results or 72. But once it goes beyond, say, 1000, you can just say 'More than 1000 objects' found to the user.

In my apps, I found that the second option is what I want. So, I limit the count() query as well, so that the counting stops when it reaches a limit. Like so:

db.datasources.find({nid: 19882}).limit(1000).count(true)

To the user, I display '1000 or more results found' if the count is 1000, otherwise, I display the exact number.

As for the first option ... I haven't thought of a neat solution yet.

Answer 3

It has to look through every field of every document for the second. You could index nid to make the count faster.