Unicorn keeps losing connection to postgres database on heroku

Question

I have setup an app on heroku using the unicorn web server. My database is setup on an external server (not on heroku). I have configured the DATABASE_URL as required. When I run heroku console I am able to successfully run queries on the server. However, when I go to a URL after unicorn has booted up, I get this error:

2014-03-14T01:44:53.820853+00:00 app[web.1]: ActiveRecord::StatementInvalid (PG::ConnectionBad: connection is closed:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
2014-03-14T01:44:53.820853+00:00 app[web.1]:                      pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
2014-03-14T01:44:53.820853+00:00 app[web.1]:                 FROM pg_attribute a LEFT JOIN pg_attrdef d
2014-03-14T01:44:53.820853+00:00 app[web.1]:                   ON a.attrelid = d.adrelid AND a.attnum = d.adnum
2014-03-14T01:44:53.820853+00:00 app[web.1]:                WHERE a.attrelid = '"currencies"'::regclass
2014-03-14T01:44:53.820853+00:00 app[web.1]:                  AND a.attnum > 0 AND NOT a.attisdropped
2014-03-14T01:44:53.820853+00:00 app[web.1]:                ORDER BY a.attnum
2014-03-14T01:44:53.820853+00:00 app[web.1]: ):
2014-03-14T01:44:53.820853+00:00 app[web.1]:   app/controllers/search_controller.rb:5:in `index'

I have configured it as per the instructions. Here is my unicorn config.

worker_processes Integer(ENV["WEB_CONCURRENCY"] || 2)
timeout 25
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end

    if defined?(ActiveRecord::Base)
      ActiveRecord::Base.connection.disconnect!
      puts 'Unicorn disconnected from database'
    end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis.quit
    Rails.logger.info('Disconnected from Redis')
  end
end

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end

  if defined?(ActiveRecord::Base)
    config = Rails.application.config.database_configuration[Rails.env]
    config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 10
    config['pool'] = ENV['DB_POOL'] || 2
    ActiveRecord::Base.establish_connection(config)
    puts 'Unicorn connected to database'
  end

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis = $redis
    Rails.logger.info('Connected to Redis')
  end
end

Why is the connection closing, even after it has made a successful connection? (I see Unicorn connected to database in the logs.

Answer 1

In Rails 3 + Resque, I had to add this:

Resque.after_fork do |job|
  ActiveRecord::Base.verify_active_connections! 
end

I'm betting something similar will work for you. Connections in the pool are timing out; and they have to be purged. There have been changes in Rails 4 around this though, see here https://github.com/socialcast/resque-ensure-connected/issues/3