Optimizing a Rails 3.0 query which returns hundreds of records

Question

Environment: Rails 3.0.6 and MySQL

I have the following two models:

class Employee < ActiveRecord::Base
  belongs_to :user
  belongs_to :manager
end

class Manager < ActiveRecord::Base
  has_many :employees
end

I have a page which shows, for a given user, all the employees under him and all their direct managers. This page typically shows tens of records but for executive level users can potentially show up to 1000 records. (For various reasons we do not want to use pagination.)

The controller method which renders the page is as follows (shown in entirety)

def manage_reports

  @tab = "home"
  @sub = "manage"

  add_breadcrumb I18n.t("home_menu.My Reports"), :manage_reports_path

  @my_reports = current_user.employees.includes(:manager).order('manager.name)      

  respond_to do |format| 
    format.html
    format.pdf { render :layout => false } if params[:format] == 'pdf'
      prawnto :filename => "employee_list.pdf", :prawn => { }
  end  
end

The view is a simple table showing columns of information for the employee and a two columns of information for the manager (name and a tag field). Neither of the two models have more than ten columns. Both are indexed.

The load time for the page ranges from << 1 second to over 15 seconds in the case of ~600 reports. After many hours of playing around with indexes, joins etc. we took another look at the New Relic performance breakdown and discovered that the time spent in the controller represents 80% of the time, whereas the DB queries actually seem to execute fairly quickly.

I have a couple of questions:

1) What is actually happening in the controller that takes 80% of the time? Is it the loading of the reports into memory? Should we not expect to be able to load that many records on to a page any quicker?

2) The manager table has a 'bio' field with a paragraph of text. How does one exclude that column when including the manager association? The Rails 3.0.x documentation seems particularly patchy on this topic. Is it even necessary to exclude that column? If the bio column is not accessed in the view, is it ever loaded into memory? I've become confused as to whether including associations in Rails 3 are now eager- or lazy-loaded. Looking at the query in the console, there is only one SQL query loading all the columns for both employee and manager.

Update:

This is the breakdown of the transaction in NewRelic. (Apologies for the formatting)

                          Time  Avg calls   Avg time (ms)  
* manage_employees          79  1           1,676  
* Manager#find              11  81          227  
* Tag#find                   9  82          185  
* Employee#find              1  2           19  
* Database SQL - SELECT     .3  1           5  
* User#find                 .1  1           3  

The view code is a simple table. Each row is as follows:

<td align="left">  <%= link_to emp.mgr.name, show_mgr_path(emp.mgr) %></td>      
<td align="center"><%= emp.mgr.lang       %></td>       
<td align="center"><%= emp.num        %></td>   
<td align="center"><%= emp.status         %></td>           
<td align="center"><%= emp.test_score     %></td>
<td align="center"><%= emp.test_date      %></td>   
<td align="center"><%= emp.serviceyrs     %></td>
<td align="left">  <%= emp.tag_list.to_s  %></td>

Answer 1

Reading your view, I think I've spotted the potential issue:

<td align="left">  <%= emp.tag_list.to_s  %></td>

It seems that your partial is executing another query to fetch the tag list. Check your logs—if that's the cae, with 600 records it's a classic N+1 problem that explains your 15 second response time.