MongoDB count is very slow

Question

I have a collections products with ~7.000.000 Books inside and a total of ~40GB mongodb 3.4 Database. Here is an example of one book document:

{ 
    "_id" : ObjectId("597f17d22be7925d9a056e82"), 
    "ean13" : "9783891491904", 
    "price" : NumberInt(2100), 
    "name" : "My cool title", 
    "author_name" : "Doe, John", 
    "warengruppe" : "HC", 
    "book_category_key" : "728",
    "keywords": ["fairy tale", "magic", "fantasy"]
    ...
}

When I query the database with limit, the timing is ok. But if I count the query (for paging), then it takes very long:

2017-08-02T13:03:16.088+0200 I COMMAND [conn74] command mydb.products command: count { count: "products", query: { book_category_key: { $in: [ "120", "130", "180", "111", "112", "140", "150", "160", "170", "190", "1AA" ] } }, readConcern: {} } planSummary: IXSCAN { book_category_key: 1 } keysExamined:1129826 docsExamined:1129825 numYields:8851 reslen:44 locks:{ Global: { acquireCount: { r: 17704 } }, Database: { acquireCount: { r: 8852 } }, Collection: { acquireCount: { r: 8852 } } } protocol:op_query 7008ms

Here is the query in nice form:

{
    count: "products",
    query: {
        book_category_key: {
            $in: ["120",
            "130",
            "180",
            "111",
            "112",
            "140",
            "150",
            "160",
            "170",
            "190",
            "1AA"]
        }
}

And this takes 7 seconds, sometimes even more (up to 20 seconds). I have an index on book_category_key:

{ 
    "v" : 2, 
    "name" : "book_category_key_1", 
    "ns" : "mydb.products", 
    "background" : true
}

Answer 1

The problem is with planSummary: IXSCAN . When count uses IXSCAN, it also does FETCH. Something like this:

"planSummary" : "IXSCAN { book_category_key: 1 }",
"execStats" : {
    "stage" : "COUNT",
    ..... 
    "inputStage" : {
        "stage" : "FETCH",
        ....
        "inputStage" : {
            "stage" : "IXSCAN",
            .....

Which in your case loads about 1/7 of your whole collection.

You can vote for https://jira.mongodb.org/browse/SERVER-17266 and linked issues, and use the proposed workaround to force COUNT_SCAN:

let cnt = 0;
for(let category of ["120",
    "130",
    "180",
    "111",
    "112",
    "140",
    "150",
    "160",
    "170",
    "190",
    "1AA"]) { cnt += db.g.count({book_category_key: category})};
print(cnt);

Which does

"planSummary" : "COUNT_SCAN { book_category_key: 1 }",
"execStats" : {
    "stage" : "COUNT",
    ...
    "inputStage" : {
        "stage" : "COUNT_SCAN"
        ....

for each category, and should be ~10 times quicker if the index fits into memory.