拆分Mongo集合（按索引）

Question

I currently have a long-running operation (running in Python+Celery) that goes through an entire Mongo collection of about 43,000,000 elements and does an analysis on the elements without making any changes to those elements.

As this collection has grown the operation has started to take longer (obviously) and now is periodically failing usually due to a timeout to a different database.

I would like to split this operation into several smaller operations--perhaps operating on just a few million elements--and I'm wondering about the best way to produce the queries that will do the splitting. I only have one index on this collection and its the _id .

The obvious answer seemed to be something like:

# This is executed in parallel on different servers
def doAnalysis(skipped,limit) 
    db.<collection>.find().skip(skipped).limit(limit)

...

# This is the parent task
elemsToAnalyze = db.<collection>.find().count()/10;
for i in range(0,10:
    doAnalysis(elemsToAnalyze * i, elemsToAnalyze)

But it turns out the .skip() takes a long time--basically just as long as actually performing the analysis! Is there a better way to do this?

Answer 1

skip() can be very slow in this kind of case. You could do range queries instead by using the last _id of the batch to query the next batch. Something like this:

db.<collection>.find({ "_id" : { $gte: prev_batch_last_id } } ).sort( { _id : 1 } ).limit(limit);

You'll have to store the last id of the batch to a variable yourself.