rails application running out of database connections

Question

Even though I have a pool of 50 in my Rails application, I also wrote a script that handles tasks periodically using the popular daemons gem. This is what it looks like:

class Responder
  def initialize
    @queue = Queue.new
  end
  # add to queue
  def produce(msg)
    @queue << msg
  end

  # take from queue
  def consume
    Thread.new do

      loop do
        sleep(1)
        if !@queue.empty?
          data = @queue.pop
          process(data)
        end
      end

    end 
  end
end

class EmailResponder < Responder
  def process(message)
    Alert.where(id: message[:id]).send_mail 
  end
end

class GeocodeResponder < Responder
  def process(message)
    Report.where(id: message[:id]).geocode_data
  end
end

class RedisListener
  def initialize(host,port)
    @host = host
    @port = port
    @email_sms = EmailResponder.new
    @geocode = GeocodeResponder.new
    # timeout so we wait for messages forever
    @redis = Redis.new(:host => @host, :port => @port, :timeout => 0)
  end

  def start_producers
    thread = Thread.new do
      @redis.subscribe('juggernaut') do |on|
        on.message do |event, msg|
          @email_sms.produce(msg)
          @geocode.produce(msg)
        end
      end
    end
  end

  def start_consumers
      @email_sms.consume
      @sidekiq.consume
  end
end

  listener = RedisListener.new('127.0.0.1', 6379)
  listener.start_producers
  listener.start_consumers

The problem is a lot of items are coming through redis so the queue builds up, and I end up using more and more database connections to the point that it crashes with postgresql max connections reached. I don't want to limit size of queue, otherwise I risk losing data that comes through redis on the fly. I'd rather let the queue grow and grow and just actually limit the database connections. How can I limit database connections in this Rails daemon (so when I use ActiveRecord objects like Alert.where(...) or Report.where(...) it will just block until db connection is free)?

I tried adding this to the script:

ActiveRecord::Base.configurations['production']['pool'] = 10

But it seems to have no effect.

Answer 1

i think its related with this topic. Connection pool issue with ActiveRecord objects in rufus-scheduler In short, active record cant share connection pool between threads

def handle_db_pools(&block)
  ActiveRecord::Base.connection_pool.with_connection &block
end

handle_db_pools do
  Alert.where(id: message[:id]).send_mail 
end

Hope this points you to the right direction