MongoDB的findandmodify队列非常慢

Question

I am using MongoDB as a queue and PHP-Queue as a way to get the data. This is a POC and I am running on an OSX machine. I am seeing very slow performance from Mongo, namely the findmodify function. I did a bunch of testing on the PHP end and PHP processing only accounts for about 5% of the time. When I fill the Mongo collection, with say, 10,000 messages, it fills very quickly, on the order of 3-5 seconds. But when I empty it, it takes about 250 seconds . Only about 10 seconds of this time is on the php side. Checking the mongod process, it never goes over about 60MB but the CPU spikes for over 90% the entire time. I have indexed the collection, and below is a sample of the message data as well as the index.

Example message(this is one of 10,000 similar messages in the queue):

{
  "_id": ObjectId("526c47d5c5008c1d5cd63ef8"),
  "payload": {
    "0": {
      "EVENT_HEADER_KEY": NumberInt(9094775),
      "event_name": "Account Change",
      "source_name": "Work",
      "event_category_name": "Complex Events",
      "EVENT_TIMESTAMP": "Aug 17 2013 12:00:00:000AM",
      "PARENT_HEADER_KEY": null,
      "year": NumberInt(2013),
      "month": NumberInt(10),
      "Company_Name": "ACME PRODUCTS, INC.",
      "Company_Email": "blabla",
      "Company_Phone": "555-555-5555",
      "First_Name": "Jon",
      "Last_Name": "Doe",
      "ID_NUMBER": "111111111",
      "created_by": "Load Job Name",
      "created_at": "Oct 18 2013 04:07:31:140PM",
      "product_analytical_category": "blabla",
      "_Event_Type": "blabla",
      "CUSTOMER_ID": "111111111"
   }
 },
  "running": false,
  "resetTimestamp": ISODate("2038-01-19T03:14:07.0Z"),
  "earliestGet": ISODate("1970-01-01T00:00:00.0Z"),
  "priority": 0,
  "created": ISODate("2013-10-26T22:53:09.440Z")
}   

Index of this collection, which seems to have been created automatically:

{
   "_id": NumberInt(1)
}

Checking the mongo.log, I can see that as I empty the queue, it only takes about 1 millisecond per message for about 70 messages, then the opid will change, and then there will be a 300-900 ms delay then it continues with the new opid at the same pace of about 1 millisecond per message. These opid changes account for about 50-100 seconds of the 250 seconds processing time, so there is still more going on.

Excerpt from the mongo.log:

**Sat Oct 26 15:15:25.189** [conn4] warning: ClientCursor::yield can't unlock b/c of recursive lock ns: test.abe top: { **opid: 20064**, active: true, secs_running: 0, op: "query", ns: "test", query: { findandmodify: "abe", query: { running: false, earliestGet: { $lte: new Date(1382825725143) } }, update: { $set: { resetTimestamp: new Date(1382825785000), running: true } }, fields: { payload: 1 }, sort: { priority: 1, created: 1 } }, client: "127.0.0.1:53045", desc: "conn4", threadId: "0x119024000", connectionId: 4, locks: { ^: "w", ^test: "W" }, waitingForLock: false, numYields: 0, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: { r: 0, w: 3 } } }

**Sat Oct 26 15:15:25.190** [conn4] warning: ClientCursor::yield can't unlock b/c of recursive lock ns: test.abe top: { **opid: 20064**, active: true, secs_running: 0, op: "query", ns: "test", query: { findandmodify: "abe", query: { running: false, earliestGet: { $lte: new Date(1382825725143) } }, update: { $set: { resetTimestamp: new Date(1382825785000), running: true } }, fields: { payload: 1 }, sort: { priority: 1, created: 1 } }, client: "127.0.0.1:53045", desc: "conn4", threadId: "0x119024000", connectionId: 4, locks: { ^: "w", ^test: "W" }, waitingForLock: false, numYields: 0, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: { r: 0, w: 3 } } }

**Sat Oct 26 15:15:25.507** [conn4] warning: ClientCursor::yield can't unlock b/c of recursive lock ns: test.abe top: { **opid: 20141**, active: true, secs_running: 0, op: "query", ns: "test", query: { findandmodify: "abe", query: { running: false, earliestGet: { $lte: new Date(1382825725501) } }, update: { $set: { resetTimestamp: new Date(1382825785000), running: true } }, fields: { payload: 1 }, sort: { priority: 1, created: 1 } }, client: "127.0.0.1:53045", desc: "conn4", threadId: "0x119024000", connectionId: 4, locks: { ^: "w", ^test: "W" }, waitingForLock: false, numYields: 0, lockStats: { timeLockedMicros: {}, timeAcquiringMicros: { r: 0, w: 3 } } }

This is basically the same throughout the entire log of these 10,000 messages. There will be a long sequence of findandmodify() which only take 1 ms per message, then the opid changes and there is a delay which can take almost a second. I have no idea if this indicates anything important, but I am new to Mongo and I am trying to find any patterns that look promising.

UPDATE:

The query checks that the field 'running' is false and it also checks that the earliestGet field is more recent than the epoch (ie 1-1-1970). I added indexes to these fields to no avail. Since these fields are the same for all messages (which are 'false' and Jan 1, 1970) in the collection, maybe that is why my indexing of them only increases the query time. I have no idea what I should do to get this to work properly. It seems that it should grab the first record it finds that is newer than Jan 1, 1970 but apparently Mongo still goes through the whole collection, which makes the query too slow to be practical. Furthermore, even when I have NO selection criteria I still get response time of 202 seconds - faster, but still unacceptable. I also still see those "yield can't unlock b/c of recursive lock ns:" messages which I thought would only show up when querying un-indexed fields.

Answer 1

You are missing a very critical index which would be used for the query and sort portion of the findAndModify command. Without that index, you are forcing each command to scan the entire collection and then sort the full result set which is inefficient. Now you said "I have indexed the collection" but you only mentioned '_id' index which is always present and cannot help you.

Recommendation: add a compound index on fields running and earliestGet at a minimum. It's possible that having an index include the sort fields may help, but since I would expect the number of matching documents for each query would be relatively small, the in memory sort may be less of a factor.

Command:

db.abe.ensureIndex({running:1, earliestGet:1})

It turned out in the comments discussion that the running, earliestGet index was not at all selective - but since you are sorting to only fetch the first matching document, the alternative is to add an index on the sort columns:

Command:

db.abe.ensureIndex({ priority: 1, created: 1 })

Answer 2

Without more detailed description what exactly are you doing during modify phase it is hard to give definitive answer. Judging from the log it seems you perform updates like this:

db.abc.findAndModify(
    query: { running: false, earliestGet: { $lte: new Date(1382825725143) } },
    update: { $set: { resetTimestamp: new Date(1382825785000), running: true } }
)

and there is no index on earliestGet field and running field. Because of low cardinality adding index on running shouldn't make a real difference, but lack of index on the earliestGet could be a real problem.

About warning: ClientCursor::yield can't unlock b/c of recursive lock ns: message you can see this question: MongoDB: Geting "Client Cursor::yield can't unlock b/c of recursive lock" warning when use findAndModify in two process instances