Rails 5 Activerecord：如何在仪表板视图中显示每个“用户”的“记录类型”

Question

I'm currently struggling to refactor some old code that shows a dashboard to a manager.

A User sends many Emails that relate to meal bookings - which are stored for auditing purposes mainly. We track the number of emails sent per user and the value of that booking, grouped by email_type

At the moment ActiveRecord is getting each email, with a where clause to filter on created_at, rails is then adding to an array which is then output to a table. This seems really inefficient and Nginx is timing out so we can't see the results.

I feel like using a mostly ActiveRecord with some groups will make this all so much simpler.

I've just added an association to user.rb as follows - because there wasn't one in place (!) and it currently isn't being utilised:

  has_many :emails, :foreign_key => "triggered_by_id"

Currently the MVC looks like so:

Model - email.rb:

  scope :sent_between, -> ( start_date, end_date ) { where("emails.created_at >= ? AND emails.created_at <= ?", start_date, end_date) }

  def self.sent_today
    sent_between(DateTime.now.beginning_of_day, DateTime.now.end_of_day)
  end

  def self.metric_hash
    {
      "venue_confirmation"                    => [0, 0],
      "enquiry_confirmation"                  => [0, 0],
      "menu_verification"                     => [0, 0],
      "amendment_information"                 => [0, 0],
      "amendment_confirmation"                => [0, 0],
      "booking_confirmation_to_venue"         => [0, 0],
      "released_confirmation_to_venue"        => [0, 0],
      "cancellation_confirmation_to_venue"    => [0, 0],
      "transfer"                              => [0, 0],
      "total"                                 => [0, 0]
    }
  end

  def self.type_for_metrics(email)
    if %w(released_confirmation_to_venue cancellation_confirmation_to_venue).include?(email.email_type)
      return "transfer" if email.booking.transferred_at
  end

    email.email_type
  end

  def self.metrics(start_date, end_date)
    metrics = {}
    totals = metric_hash
    observed_bookings = Set.new

     Email.includes(:booking).select(:id, :email_type, :triggered_by_id, :booking_id).sent_between(start_date, end_date).references(:booking).select(:booking_total).find_each do |email|

      email_type = type_for_metrics(email)

      if totals.has_key?(email_type)
        metrics[email.triggered_by_id] ||= metric_hash
        metrics[email.triggered_by_id][email_type][0] += 1
        metrics[email.triggered_by_id][email_type][1] += email.booking.booking_total
        metrics[email.triggered_by_id]["total"][0] += 1
        metrics[email.triggered_by_id]["total"][1] += email.booking.booking_total
        totals[email_type][0] += 1
        totals[email_type][1] += email.booking.booking_total
        totals["total"][0] += 1

        # Only count the each booking once for the total value
        unless observed_bookings.include?(email.booking_id)
          totals["total"][1] += email.booking.booking_total
          observed_bookings << email.booking_id
        end
      end
    end

    results = metrics.inject({}) do |memo, row|
      if row[1]["total"][0] > 0
        if row[0]
          user = User.find(row[0])
          memo[user.name] = row[1]
        else
          memo["Sent Before Tracking"] = row[1]
        end
        memo
      end
    end
    results["Total"] = totals

    results
  end

index_controller.rb:

  def metrics
      @start = ( params[:start] && Time.zone.parse(params[:start]) ) || DateTime.now.start_of_period
      @end   = ( params[:end] && Time.zone.parse(params[:end]) ) || DateTime.now.end_of_period
      @email_metrics = Email.metrics(@start, @end)
  end

_metrics.html.erb

<h2>Emails Sent</h2>

  <table>
    <thead>
      <tr>
        <th></th>
        <th title="New Enquiry">New</th>
        <th title="Menu Confirmation">Menu</th>
        <th title="Operator Confirmation">Confirm</th>
        <th title="Released Enquiry">Released</th>
        <th title="Cancelled Booking">Cancelled</th>
        <th title="Amendment Information">Amend Info</th>
        <th title="Amendment Confirmation">Amend Confirm</th>
        <th title="Transferred">Transfer</th>
        <th>Total</th>
      </tr>
    </thead>
    <tbody>
      <% @email_metrics.each do |key, metrics| %>
        <tr>
          <th rowspan="2"><%= key %></th>
          <td><%= metrics["venue_confirmation"][0] %></td>
          <td><%= metrics["menu_verification"][0] %></td>
          <td><%= metrics["booking_confirmation_to_venue"][0] %></td>
          <td><%= metrics["released_confirmation_to_venue"][0] %></td>
          <td><%= metrics["cancellation_confirmation_to_venue"][0] %></td>
          <td><%= metrics["amendment_information"][0] %></td>
          <td><%= metrics["amendment_confirmation"][0] %></td>
          <td><%= metrics["transfer"][0] %></td>
          <td><%= metrics["total"][0] %></td>
        </tr>
        <tr>
          <td><%= number_to_currency metrics["venue_confirmation"][1] %></td>
          <td><%= number_to_currency metrics["menu_verification"][1] %></td>
          <td><%= number_to_currency metrics["booking_confirmation_to_venue"][1] %></td>
          <td><%= number_to_currency metrics["released_confirmation_to_venue"][1] %></td>
          <td><%= number_to_currency metrics["cancellation_confirmation_to_venue"][1] %></td>
          <td><%= number_to_currency metrics["amendment_information"][1] %></td>
          <td><%= number_to_currency metrics["amendment_confirmation"][1] %></td>
          <td><%= number_to_currency metrics["transfer"][1] %></td>
          <td>N/A</td>
        </tr>
      <% end %>
    </tbody>
  </table>
</div>

My feeling is that I'd start with User.emails.etc... but I'm getting stuck with the multiple group_by and the massively complicated array currently in place.

Here's a screenshot of how it should look. Dev environment only has one User currently.

Screenshot of how the data should look

Answer 1

I have came up a solution.

  scope :sent_between, -> ( start_date, end_date ) { where("emails.created_at >= ? AND emails.created_at <= ?", start_date, end_date) }

  def self.sent_today
    sent_between(DateTime.now.beginning_of_day, DateTime.now.end_of_day)
  end

  def self.metric_hash
    {
        "venue_confirmation"                    => [0, 0],
        "enquiry_confirmation"                  => [0, 0],
        "menu_verification"                     => [0, 0],
        "amendment_information"                 => [0, 0],
        "amendment_confirmation"                => [0, 0],
        "booking_confirmation_to_venue"         => [0, 0],
        "released_confirmation_to_venue"        => [0, 0],
        "cancellation_confirmation_to_venue"    => [0, 0],
        "transfer"                              => [0, 0],
        "total"                                 => [0, 0]
    }
  end

  def self.type_for_metrics(email)
    if %w(released_confirmation_to_venue cancellation_confirmation_to_venue).include?(email.email_type)
      return "transfer" if email.booking.transferred_at
    end

    email.email_type
  end

  def self.metrics(start_date, end_date)
    metrics = {}
    totals = metric_hash
    observed_bookings = {}

    Email.includes(:booking).select(:id, :email_type, :triggered_by_id, :booking_id).sent_between(start_date, end_date).
        references(:booking).select(:booking_total).find_each do |email|

      email_type = type_for_metrics(email)

      if totals[email_type]
        metrics[email.triggered_by_id] ||= metric_hash
        metrics[email.triggered_by_id][email_type][0] += 1
        metrics[email.triggered_by_id][email_type][1] += email.booking.booking_total
        metrics[email.triggered_by_id]["total"][0] += 1
        metrics[email.triggered_by_id]["total"][1] += email.booking.booking_total
        totals[email_type][0] += 1
        totals[email_type][1] += email.booking.booking_total
        totals["total"][0] += 1

        # Only count the each booking once for the total value
        unless observed_bookings[email.booking_id]
          totals["total"][1] += email.booking.booking_total
          observed_bookings[email.booking_id] = true
        end
      end
    end

    users = users = User.select(:id,:name).where(id: metrics.keys).group_by(&:id).transform_values{ |value| value.first }
    results = metrics.inject({}) do |memo, row|
      if row[1]["total"][0] > 0
        if row[0]
          user = users[row[0]]
          memo[user.name] = row[1]
        else
          memo["Sent Before Tracking"] = row[1]
        end
        memo
      end
    end
    results["Total"] = totals

    results
  end

What I have done here is I changed the type of observed_bookings to hash. Because hashes has a blazing fast look up and inserting new item to hash is O(1) in your scenario. In your scenario, you make a look up and you want the data structure to be uniq. So hash is your friend.

I have changed totals.has_key?(email_type) to totals[email_type] . Again here we have blazing look up which O(1).

And lastly, because of the fact that user ids are actually metrics keys, so with just only query we can fetch the users from database and convert the result result to hash.

And yes, users are hash and user id is the key of hash. So again we have a blazing fast lookup which is O(1).

I think this query should give what you expect in terms of response time.

Answer 2

I agree with you that just retrieving all emails, iterating over them and building the result set manually seems completely counterproductive.

You will not be able to fix this with a single query, but instead of fetching 10000 records you would be able to solve it with a few (faster queries) and let the database do its work.

I will give some examples that hopefully get you started.

Fetching all types for a given period:

Email.sent_between(start_date, end_date).group(:email_type).count 

Fetching the amount of mails per user, in a given period

Email.sent_between(start_date, end_date).group(:triggered_by_id).count 

This returns two things: the users and their count. Use those users to get the result per user:

Email.where(trigger_by_id: user_id).sent_between(start_date, end_date).group(:email_type).count 

The booking-total seems a little harder, but I think something like this should work:

Email.includes(:booking).sent_between(start_date, end_date).references(:booking).group(:email_type).sum(:booking_total) 

I understand this is not a complete solution, you still have to compose your "result-hash" but this should fetch your raw data a lot quicker.