批量插入Mongo-红宝石

Question

I am new to Ruby and Mongo and am working with twitter data. I'm using Ruby 1.9.3 and Mongo gems.

I am querying bulk data out of Mongo, filtering out some documents, processing the remaining documents (inserting new fields) and then writing new documents into Mongo.

The code below is working but runs relatively slow as I loop through using .each and then insert new documents into Mongo one at a time.

My Question: How can this be structured to process and insert in bulk?

cursor = raw.find({'user.screen_name' => users[cur], 'entities.urls' => []},{:fields => params})

cursor.each do |r| 
  if r['lang'] == "en"
    score = r['retweet_count'] + r['favorite_count']
    timestamp = Time.now.strftime("%d/%m/%Y %H:%M")

    #Commit to Mongo
    @document = {:id => r['id'],
                :id_str => r['id_str'],
                :retweet_count => r['retweet_count'],
                :favorite_count => r['favorite_count'],
                :score => score,    
                :created_at => r['created_at'],
                :timestamp => timestamp,
                :user => [{:id => r['user']['id'],
                           :id_str => r['user']['id_str'],
                           :screen_name => r['user']['screen_name'],
                          }
                         ]
                }
    @collection.save(@document)   
    end #end.if
end #end.each

Any help is greatly appreciated.

Answer 1

In your case there is no way to make this much faster. One thing you could do is retrieve the documents in bulks, processing them and the reinserting them in bulks, but it would still be slow.

To speed this up you need to do all the processing server side, where the data already exist.

You should either use the aggregate framework of mongodb if the result document does not exceed 16mb or for more flexibility but slower execution (much faster than the potential your solution has) you can use the MapReduce framework of mongodb

Answer 2

What exactly are you doing? Why not going pure ruby or pure mongo (well that's ruby too) ? and Why do you really need to load every single attribute?

What I've understood from your code is you actually create a completely new document, and I think that's wrong.

You can do that with this in ruby side:

cursor = YourModel.find(params)

cursor.each do |r|
    if r.lang == "en"
        r.score = r.retweet_count + r.favorite_count
        r.timestamp = Time.now.strftime("%d/%m/%Y %H:%M")
        r.save
    end #end.if
end #end.each

And ofcourse you can import include Mongoid::Timestamps in your model and it handles your created_at , and updated_at attribute (it creates them itself)

in mongoid it's a little harder first you get your collection with use my_db then the next code will generate what you want

db.models.find({something: your_param}).forEach(function(doc){
    doc.score = doc.retweet_count + doc.favorite_count
    doc.timestamp = new Timestamp()
    db.models.save(doc)
    }
);

I don't know what was your parameters, but it's easy to create them, and also mongoid really do lazy loading, so if you don't try to use an attribute, it won't load that. You can actually save a lot of time not using every attribute. And these methods, change the existing document, and won't create another one.