如何在Rails中批量编写查询？

Question

I have a users table with 800 000 records. I created a new field called token in users table. for all the new users token is getting populated. for existing users to populate the token i wrote a rake task with following code. i feel this is not work for these many records in production environment. How to rewrite these queries with batches or some other way of writing the queries

users = User.all
users.each do |user|
 user.token = SecureRandom.urlsafe_base64(nil, false)
 user.save
end

Answer 1

How you want to proceed depends on different factors: is validation important for you when executing this? Is time an issue? If you don't care about validations, you may generate raw SQL queries for each user and then execute them at once, otherwise you have options like ActiveRecord transactions:

User.transaction do
  users = User.all
  users.each do |user|
    user.update(token: SecureRandom.urlsafe_base64(nil, false))
  end
end

This would be quicker than your rake task, but still would take some time, depending on the number of users you want to update at once.

Answer 2

lower_limit = User.first.id
upper_limit = 30000
while true

  users = User.where('id >= ? and  id< ?',lower_limit,upper_limit)
  break if users.empty?
  users.each do |user|
    user.update(token: SecureRandom.urlsafe_base64(nil, false))
  end
  lower_limit+=30000
  upper_limit+=30000
end

Answer 3

I think that the best option for you is to use find_each or transactions .

Doc for find_each:

Looping through a collection of records from the database (using the ActiveRecord::Scoping::Named::ClassMethods#all method, for example) is very inefficient since it will try to instantiate all the objects at once.

In that case, batch processing methods allow you to work with the records in batches, thereby greatly reducing memory consumption.

The find_each method uses find_in_batches with a batch size of 1000 (or as specified by the :batch_size option).

Doc for transaction:

Transactions are protective blocks where SQL statements are only permanent if they can all succeed as one atomic action

In case that you care about memory, because you are bringnig all the 800k of users in memory, the User.all.each will instantiate the 800k objects consuming a lot of memory so my approach will be:

User.find_each(batch_size: 500) do |user|
  user.token = SecureRandom.urlsafe_base64(nil, false)
  user.save
end

In this case, it only instantiate 500 users instead of 1000 that is the default batch_size .

If you still want to do it in only one transaction to the database, you can use the answer of @Francesco

Answer 4

The common mistake is instantiating model instance without need. While AR instantiating is not cheap. You can try this naive code:

BATCH_SIZE = 1000
while true
  uids = User.where( token: nil ).limit( BATCH_SIZE ).pluck( :id )
  break if uids.empty?
  ApplicationRecord.transaction do
    uids.each do |uid|
      # def urlsafe_base64(n=nil, padding=false)
      User
        .where( id: uid )
        .update_all( token: SecureRandom.urlsafe_base64 )
    end
  end
end

Next option is to use native DB's analog for SecureRandom.urlsafe_base64 and run one query like:

UPDATE users SET token=db_specific_urlsafe_base64 WHERE token IS NULL

If you won't find the analog, you can prepopulate temp table (like PostgreSQL 's' COPY command) from precalculated CSV file(id, token=SecureRandom.urlsafe_base64) and run one query like:

UPDATE users SET token=temp_table.token
FROM temp_table
WHERE (users.token IS NULL) AND (users.id=temp_table.id)

But in fact you need no to fill token on existing users because of:

i am using "token" for token based authentication in rails – John

You have to check if user's token is NULL (or expired) and redirect him to login form. It's common way and it will save your time.