Yet another mongodb indexOnly

Question

I've spent the better part of this morning re-reading MongoDB docs , blog s and other answers on the stack her and I'm still missing something which I hope is painfully obvious to others.

EDIT: I've changed the scheme of the document to not have sub-documents (metadata.*) and am still having problems with the index not being covered. I've dropped the existing indexes and re-indexed with new ones:

So not I've got:

[
    {
        "v" : 1,
        "key" : {
                "_id" : 1
        },
        "name" : "_id_",
        "ns" : "test.daily"
    },
    {
        "v" : 1,
        "key" : {
                "host" : 1,
                "cid" : 1,
                "title" : 1,
                "urls" : 1,
                "global" : -1,
                "current" : -1,
                "total" : -1
        },
        "name" : "byHostTotals",
        "ns" : "test.daily"
    },
    {
        "v" : 1,
        "key" : {
                "host" : 1,
                "cid" : 1,
                "title" : 1,
                "urls" : 1,
                "total" : -1,
                "global" : -1,
                "current" : -1
        },
        "name" : "byHostCurrents",
        "ns" : "test.daily"
    }
]

And given this query:

db.daily.find({'host': 'example.com'}, {'_id': 0, 'cid': 1, 'title': 1, 'current': 1}).hint("byHostCurrents").sort({'current': -1}).limit(10).explain()

is not showing up as being covered by the index named "byHostCurrent":

{
    "clauses" : [
        {
            "cursor" : "BtreeCursor byHostCurrents",
            "isMultiKey" : true,
            "n" : 10,
            "nscannedObjects" : 1090,
            "nscanned" : 1111,
            "scanAndOrder" : true,
            "indexOnly" : false,
            "nChunkSkips" : 0,
            "indexBounds" : {
                "host" : [
                        [
                            "example.com",
                            "example.com"
                        ]
                ],
                "cid" : [
                    [
                        {
                            "$minElement" : 1
                        },
                        {
                            "$maxElement" : 1
                        }
                    ]
                ],
                "title" : [
                    [
                        {
                            "$minElement" : 1
                        },
                        {
                            "$maxElement" : 1
                        }
                    ]
                ],
                "total" : [
                    [
                        {
                            "$maxElement" : 1
                        },
                        {
                            "$minElement" : 1
                        }
                    ]
                ],
                "global" : [
                    [
                        {
                            "$maxElement" : 1
                        },
                        {
                            "$minElement" : 1
                        }
                    ]
                ],
                "current" : [
                    [
                        {
                            "$maxElement" : 1
                        },
                        {
                            "$minElement" : 1
                        }
                    ]
                ]
            }
        },
        {
            "cursor" : "BtreeCursor ",
            "isMultiKey" : false,
            "n" : 0,
            "nscannedObjects" : 0,
            "nscanned" : 0,
            "scanAndOrder" : true,
            "indexOnly" : false,
            "nChunkSkips" : 0,
            "indexBounds" : {
                "host" : [
                        [
                                "usatoday.com",
                                "usatoday.com"
                        ]
                ],
                "cid" : [
                    [
                        {
                                "$minElement" : 1
                        },
                        {
                                "$maxElement" : 1
                        }
                    ]
                ],
                "title" : [
                    [
                        {
                                "$minElement" : 1
                        },
                        {
                                "$maxElement" : 1
                        }
                    ]
                ],
                "total" : [
                    [
                        {
                                "$maxElement" : 1
                        },
                        {
                                "$minElement" : 1
                        }
                    ]
                ],
                "global" : [
                    [
                        {
                                "$maxElement" : 1
                        },
                        {
                                "$minElement" : 1
                        }
                    ]
                ],
                "current" : [
                    [
                        {
                                "$maxElement" : 1
                            },
                            {
                                "$minElement" : 1
                            }
                        ]
                    ]
                }
            }
        ],
    "cursor" : "QueryOptimizerCursor",
    "n" : 10,
    "nscannedObjects" : 1090,
    "nscanned" : 1111,
    "nscannedObjectsAllPlans" : 1090,
    "nscannedAllPlans" : 1111,
    "scanAndOrder" : false,
    "nYields" : 8,
    "nChunkSkips" : 0,
    "millis" : 9,
    "server" : "ubuntu:27017",
    "filterSet" : false
}

MongoDB version is: 2.6.3.

Answer 1

So here's the skinny...

When the Query:

db.daily.find({'host': 'example.com'}, {'_id': 0, 'cid': 1, 'title': 1, 'current': 1}).hint("byHostCurrents").sort({'current': -1}).limit(10);

If I didn't have the .sort() in there it would use the index, however since I'm using sort NOW the ORDER of the Indexed fields become important.

For the above query to use an index, I'd need to make a new index like this:

db.daily.ensureIndex({'current': -1, 'host': 1, 'cid': 1, 'title': 1}); 

Now with this index in place, we'll get indexOnly: true since we're looking down the total currents in reverse order (descending) and we'll only have to scan as many entries in the index as is needed to meet the 'host' = 'example.com' and limit requirements.

So in total I had to have 4 additional indexes to support my queries:

1. one to find the content ids with the most current people on it (the above index)
2. one to find the content ids that have had the most people on it (like the above but totals: -1 rather than current:-1)
3. one to find content by host sorted by current (see index below) and,

4. one to find content by host sorted by total (like the one below)

   db.daily.ensureIndex({'host': 1, 'current': -1, 'cid': 1, 'title': 1});

So the MongoDB docs are not very clear on their explanation of these things especially when looking at the sort issue. What they lack to say is if you are going to use a sort, you have to include the prefix fields after your equality query or to include all the prefix fields.

For example given my original index from my question:

db.daily.ensureIndex({"host" : 1, "cid" : 1, "title" : 1, "urls" : 1, "global" : -1, "current" : -1, "total" : -1});

If I wanted a query to be covered by the index then I'd have to change from this:

db.daily.find({'host': 'example.com'}, {'_id': 0, 'cid': 1, 'title': 1, 'current': 1}).hint("byHostCurrents").sort({'current': -1}).limit(10);

To This:

db.daily.find({'host': 'example.com'}, {'_id': 0, 'cid': 1, 'title': 1, 'current': 1}).hint("byHostCurrents").sort({'cid':1, 'title':1, 'urls': 1, 'global: 1, 'current': -1}).limit(10);

which is not what I wanted.

Hope this helps someone in the future.