What is the best way to store a multi-dimensional counter_cache?

Question

In my application I have a search_volume.rb model that looks like this:

search_volume.rb :

class SearchVolume < ApplicationRecord
   # t.integer "keyword_id"
   # t.integer "search_engine_id"
   # t.date "date"
   # t.integer "volume"
  belongs_to :keyword
  belongs_to :search_engine
end

keyword.rb :

class Keyword < ApplicationRecord
 has_and_belongs_to_many :labels
 has_many :search_volumes

end

search_engine.rb :

class SearchEngine < ApplicationRecord
  belongs_to :country
  belongs_to :language
end

label.rb :

class Label < ApplicationRecord
 has_and_belongs_to_many :keywords
 has_many :search_volumes, through: :keywords

end

On the label#index page I am trying to show the sum of search_volumes for the keywords in each label for the last month for the search_engine that the user has cookied. I am able to do this with the following:

<% @labels.each do |label| %>
  <%= number_with_delimiter(label.search_volumes.where(search_engine_id: cookies[:search_engine_id]).where(date: 1.month.ago.beginning_of_month..1.month.ago.end_of_month).sum(:volume)) %>
<% end %>

This works well but I have the feeling that the above is very inefficient. With the current approach I also dind it difficult to do operations on search volumes. Most of the time I just want to know about last month's search volume.

Normally I would create a counter_cache on the keywords model to keep track of the latest search_volume, but since there are dozens of search_engines I would have to create one for each, which is also inefficient.

What's the most efficient way to store last month's search volume for all the different search engines separately?

Answer 1

First of all, you can optimize your current implementation by doing one single request for all involved labels like so:

# models
class SearchVolume < ApplicationRecord
  # ...

  # the best place for your filters!
  scope :last_month, -> { where(date: 1.month.ago.beginning_of_month..1.month.ago.end_of_month) }
  scope :search_engine, ->(search_engine_id) { where(search_engine_id: search_engine_id) }
end

class Label < ApplicationRecord
  # ...

  # returns { label_id1 => search_volumn_sum1, label_id2 => search_volumn_sum2, ... }
  def self.last_month_search_volumes_per_label_report(labels, search_engine_id:)
    labels.
      group(:id).
      left_outer_joins(:search_volumes).
      merge(SearchVolume.last_month.search_engine(search_engine_id)).
      pluck(:id, 'SUM(search_volumes.volume)').
      to_h
  end
end

# controller
class LabelsController < ApplicationController
  def index
    @labels = Label.all

    @search_volumes_report = 
      Label.last_month_search_volumes_per_label_report(
        @labels, search_engine_id: cookies[:search_engine_id]
      )
  end
end

# view
<% @labels.each do |label| %>
  <%= number_with_delimiter(@search_volumes_report[label.id]) %>
<% end %>

Please note that I have not tested it with the same architecture, but with similar models I have on my local machine. It may work by adjusting a few things.

My proposed approach still is live requesting the database. If you really need to store values somewhere because you have very large datasets, I suggest two solutions:
- using materialized views you could refresh each month ( scenic gem offers a good way to handle views in Rails application: https://github.com/scenic-views/scenic )
- implementing a new table with standard relations between models, that could store your calculations by ids and months and whose you could populate each month using rake tasks, then you would simply have to eager load your calculations

Please let me know your feedbacks!