Heroku Postgres Connections on Rails 3

Question

Been reading up on it, but not sure I totally understand how connections to Postgres work in Rails. I have up to 5 web dynos, each running 5 processes (Unicorn), so that would be up to 25 connections, and then up to 50 worker dynos through delayed_job, so that would be 75.

But when the worker queue grows and I'm running all 50 worker dynos, it seems like I'm hitting the Postgres connection limit of 128 and I imagine that can't be good.

Is it that the web dynos can open multiple connections per process? Can the workers do the same?

If it's to do with the pool database setting, for Unicorn is that the pool of connections per worker process or per dyno? That's not clear to me (as per comments with Corey), although it seems like per Unicorn worker process.

Is there a way to set the delayed_job worker dynos pool setting? I'm guessing that's the default of 5 as well.

Any good resources or solutions would be really helpful. Seems like PGbouncer is one option, or setting the pool from the default 5 to something lower like 1 or 2.

EDIT: I mistakenly read the NewRelic logs as ms per transaction, but it is total ms for that category, so it's not slowing Postgres down, but hitting the connection limit is still not ideal.

Answer 1

A single instance of Rails can handle multiple incoming requests using threads, despite the GIL, so the default configuration is to open multiple connections to the database (one per thread) and manage access to them using a connection pool. By default, Rails is configured to maintain up to 5 connections in this pool, but you can tune the size based on your needs.

Normally I would provide this information in the answer, but there are a lot of factors to consider, so I'm going to instead provide links to two documents that you should review:

1. Concurrency and Database Connections in Ruby with ActiveRecord
2. Deploying Rails Applications with Unicorn

Finally, I'll leave you with some hard won advice: Ditch Unicorn and DelayedJob in favor of Puma and Sidekiq . Puma + Sidekiq are both multi-threaded and developed with performance in mind. Puma is the default application server in Rails 5, and Heroku recommends it as well. Sidekiq is the de facto background job processing library in the Ruby community and enjoys wide support and commercial backing.