Sidekiq & Clock Background Task On Redis, Rails, Heroku Only Updating Some

Question

I am running a background task on many records with Rails, Sidekiq, Redis, and Clockwork on Heroku, but I keep getting this error on many of the records it's trying to update with an API:

ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds

Here are my files:

Unicorn:

worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)
timeout 15
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10 # seconds
    config['pool']            = ENV['DB_POOL'] || 5
    ActiveRecord::Base.establish_connection(config)
  end
end

Database:

production:
  adapter: postgresql
  encoding: unicode
  database: production
  pool: 25
  timeout: 10000

Clock:

every(24.hours, 'Update') do
  sites = Site.order(:id).pluck(:id)
  Site.each do |site|
    UpdatePosts.perform_async(site)
  end
end

UpdatePosts

class UpdatePosts
    include Sidekiq::Worker
    sidekiq_options retry: false

    def perform(site_id)
    ...
    end
end

Answer 1

The error you're seeing is caused by more threads trying to obtain a connection from the ActiveRecord connection pool than there are connections in the pool. When a thread asks for a connection, but one isn't free within the 5 second timeout, the error you see is raised.

In Sidekiq, there are a larger number of worker threads than the default ActiveRecord pool size. This causes errors like this under load.

You can adjust the DB pool size to match the number of Sidekiq threads in your config/initializers/sidekiq.rb file using code like this:

Sidekiq.configure_server do |config|
  if(database_url = ENV['DATABASE_URL'])
    pool_size = Sidekiq.options[:concurrency] + 2
    ENV['DATABASE_URL'] = "#{database_url}?pool=#{pool_size}"
    ActiveRecord::Base.establish_connection
  end
end

Although you have attempted to increase the pool_size in your database.yml file, this configuration is overwritten on deploy in Heroku and driven entirely by the DATABASE_URL environment variable.