mongodb slow query performance

Question

I have the following mongo test cluster in place

1. No of shards -2
2. No of config server -1
3. mongos instances -2
4. Replication not enabled

I have around 41 million records split across the shards, I have defined a compound index {field1:1,field2:1,field3:1}, my queries are of the form (field=1 and field2 between x and y), I expected the compound index to be useful for these queries, however the query response time is around 8 sec for the query I described. I am specifying only the fields of interest when I execute find.

Mongos is installed on the machine from where I execute the query and I am using java to do the querying.

Can someone throw some light on the possible reasons, why this query takes such a long time? I would be happy to provide more information if required.

The following is the output of explain command

{
    "indexBounds": {
        "LOGIN_ID": [
            [
                {
                    "$minElement": 1
                }, 
                {
                    "$maxElement": 1
                }
            ]
        ], 
        "LOGIN_TIME": [
            [
                1262332800000, 
                1293782400000
            ]
        ]
    }, 
    "nYields": 7, 
    "millisShardTotal": 7410, 
    "millisShardAvg": 7410, 
    "numQueries": 1, 
    "nChunkSkips": 0, 
    "shards": {
        "server1:27017": [
            {
                "nYields": 7, 
                "nscannedAllPlans": 1769804, 
                "allPlans": [
                    {
                        "cursor": "BtreeCursor LOGIN_TIME_1_LOGIN_ID_1", 
                        "indexBounds": {
                            "LOGIN_ID": [
                                [
                                    {
                                        "$minElement": 1
                                    }, 
                                    {
                                        "$maxElement": 1
                                    }
                                ]
                            ], 
                            "LOGIN_TIME": [
                                [
                                    1262332800000, 
                                    1293782400000
                                ]
                            ]
                        }, 
                        "nscannedObjects": 1763903, 
                        "nscanned": 1763903, 
                        "n": 14081
                    }, 
                    {
                        "cursor": "BasicCursor", 
                        "indexBounds": {}, 
                        "nscannedObjects": 5901, 
                        "nscanned": 5901, 
                        "n": 0
                    }
                ], 
                "millis": 7410, 
                "nChunkSkips": 0, 
                "server": "server2:27017", 
                "n": 14081, 
                "cursor": "BtreeCursor LOGIN_TIME_1_LOGIN_ID_1", 
                "oldPlan": {
                    "cursor": "BtreeCursor LOGIN_TIME_1_LOGIN_ID_1", 
                    "indexBounds": {
                        "LOGIN_ID": [
                            [
                                {
                                    "$minElement": 1
                                }, 
                                {
                                    "$maxElement": 1
                                }
                            ]
                        ], 
                        "LOGIN_TIME": [
                            [
                                1262332800000, 
                                1293782400000
                            ]
                        ]
                    }
                }, 
                "scanAndOrder": false, 
                "indexBounds": {
                    "LOGIN_ID": [
                        [
                            {
                                "$minElement": 1
                            }, 
                            {
                                "$maxElement": 1
                            }
                        ]
                    ], 
                    "LOGIN_TIME": [
                        [
                            1262332800000, 
                            1293782400000
                        ]
                    ]
                }, 
                "nscannedObjectsAllPlans": 1769804, 
                "isMultiKey": false, 
                "indexOnly": false, 
                "nscanned": 1763903, 
                "nscannedObjects": 1763903
            }
        ]
    }, 
    "n": 14081, 
    "cursor": "BtreeCursor LOGIN_TIME_1_LOGIN_ID_1", 
    "oldPlan": {
        "cursor": "BtreeCursor LOGIN_TIME_1_LOGIN_ID_1", 
        "indexBounds": {
            "LOGIN_ID": [
                [
                    {
                        "$minElement": 1
                    }, 
                    {
                        "$maxElement": 1
                    }
                ]
            ], 
            "LOGIN_TIME": [
                [
                    1262332800000, 
                    1293782400000
                ]
            ]
        }
    }, 
    "numShards": 1, 
    "clusteredType": "ParallelSort", 
    "nscannedAllPlans": 1769804, 
    "nscannedObjectsAllPlans": 1769804, 
    "millis": 7438, 
    "nscanned": 1763903, 
    "nscannedObjects": 1763903
}

A sample document in my db is as follows

    {
    "_id" : ObjectId("52d5192c1a45f84e48c24e2f"),
    "LOGIN_ID" : <loginId>,
    "LOGIN_TIME" : NumberLong("1372343932000"),
    "BUSINESS_ID" : <businessId>,
    "USER_ID" : <userid>,
    "EMAIL" : "a@b.com",
    "SITE_POD_NAME" : "x",
    "USER_AGENT" : "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML. like      Gecko) Chrome/26.0.1410.43 Safari/537.31"
    }

There are some other fields in the above doc which I cannot expose outside, but its a simple key value of string and string

This is how I query the db

    DBObject dbObject = new BasicDBObject("BUSINESS_ID", businessId)
        .append("LOGIN_TIME",
new BasicDBObject("$gte",start).append("$lt", end))
    .append("LOGIN_TYPE", loginType);

    long startTime = System.currentTimeMillis();
    DBObject keys = new BasicDBObject("LOGIN_TIME", 1);
DBCursor find = collection.find(dbObject, keys);

int count = 0;
while (find.hasNext()) {
    find.next();
    count++;
}
long endTime = System.currentTimeMillis();

Mongo DB version is 2.4.9. Appreciate any help.

Answer 1

I see a following spots which could head into finding more about exact issue:

1. What is login_time and what does the numbers in the query range actually mean? The range looks quite wide by numeric difference. May be you filter criteria is vey wide-ranged? This is also indicative from the "nscanned" from the explain plan.

2. I see the index is on login_time and login_id, where as your query is on login_time and login_type. Just high-lighting that although you are using index, your query criteria is wide enough to cover a much larger index range and since the second criteria of login_type is not part of the index, query would need to fetch all "nscanned" documents to determine if it a valid record for this query.