5M和150M网页的Mongo数据库真的很慢

Question

I'm storing a collection of webpages in a mongodb, around 150M webpages. Each page of different size. The only transaction I want to do it to retrieve pages using their id (not mongodb default _id). However, it takes really too long time to get results and I didn't manage to retrieve any document yet. However, use db.collection.findOne() works perfectly. Hence, I indexed a subset of 5M webpages for testing and repairing. When issue a query against this db db.collection.find("id":"aw-000") , it takes 4 minutes or more to get a document.

I tried db.runCommand({compact: 'collection'}) and db.runCommand({compact: 'collection'}) m but they didn't help!

When I checked the logs under var/log/mongodb/mongod.log (that should contains any query took more than 100ms), I found this:

655163:2017-07-16T14:05:37.231+0300 I COMMAND  [ftdc] serverStatus was very slow: { after basic: 0, after asserts: 0, after connections: 0, after extra_info: 310, after globalLock: 310, after locks: 310, after network: 310, after opcounters: 310, after opcountersRepl: 310, after storageEngine: 310, after tcmalloc: 310, after wiredTiger: 310, at end: 1220 }

However, I don't know how to benefit from such logs.

Is there a way to make my db more efficient?

Answer 1

As pointed by Neil Lunn in the comments above. I found the easiest solution is to create the db from scratch, while using _id as my id field name instead of "id" . _id has an index by default and the only type of queries will be issued against this index is retrieving by id.

So, the program (any program that's used to create the index) will insert the object as following:

db.collection.insert( { _id: "aw-000", page: "...", .... } )

instead of:

db.collection.insert( { id: "aw-000", page: "...", .... } )