Rake task too slow

Question

I have a rake task which accumulate all info from several tables into one, each table is about 200 000 rows:

task :migrate_all_info_tables_to_one => :environment do
    profiles = Profile.all
    profiles.each do |profile|
      basic_info_items = profile.basic_info_items
      basic_info_items.each do |item|
        info = InfoItem.find_or_initialize_by_kind_and_value_and_link_and_description(:kind => 'info', :value => item.parameter, :link => item.link != 'none' ? item.link : nil, :description => item.value)
        profile.info_items << info
      end

      cities = profile.cities
      cities.each do |city|
        city = InfoItem.find_or_initialize_by_kind_and_value_and_link_and_description(:kind => 'city', :value => city.title, :link => city.link != 'none' ? city.link : nil, :description => city.description)
        profile.info_items << city
      end

      histories = profile.histories
      histories.each do |item|
        info = InfoItem.find_or_initialize_by_kind_and_value_and_link_and_description(:kind => 'history', :value => item.year, :link => item.link != 'none' ? item.link : nil, :description => item.event)
        profile.info_items << info
      end

      contacts = profile.contacts
      contacts.each do |item|
        info = InfoItem.find_or_initialize_by_kind_and_value_and_link_and_description(:kind => 'contact', :value => item.parameter, :link => item.link != 'none' ? item.link : nil, :description => item.value)
        profile.info_items << info
      end
    end
  end

I added all necessary indexes in tables (using mysql 5.1.6), form mysql each separate query take 0.00 sec, from ruby console too. But when I run this rake task it processing only 100 profiles records per minute, why or maybe I am doing smth wrong?

PS: I do this because I made ​​a mistake in my project database architecture.

Answer 1

To make it more efficient you can eager load the profile relations you use in the script, use :

Profile.includes(:basic_info_items, :cities, :histories, :contacts).all

instead of Profile.all .

You may also want to use Profile.find_each instead of Profile.all to avoid loading all the profile in memory.

Answer 2

If you have related records on Profile, and are accessing them as you loop over all the profiles, ActiveRecord might be doing a separate select for each one, complete with BEGIN and COMMIT. This will slow things down even if each select is very fast.

http://api.rubyonrails.org/classes/ActiveRecord/Associations/ClassMethods.html

Have a look at the section on eager loading associations.