Rails3 active record pool and Sidekiq multi-thread

Question

I am using sidekiq with rails3. Sidekiq runs 25 threads default. I would like to increase multi-thread limit, I have done this by changing sidekiq.yml.

So, what is the relation between pool value in database.yml and sidekiq multi-thread. What is the maximun value of mysql pool. Is it depends on server memory?

sidekiq.yml

:verbose: true
:concurrency:  50
:pool: 50
:queues:
  - [queue_primary, 7]
  - [default, 5]
  - [queue_secondary, 3]

database.yml

production:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: db_name
  pool: 50
  username: root
  password: root
  socket: /var/run/mysqld/mysqld.sock

Answer 1

Each Sidekiq job executes in one of up to 50 threads with your configuration. Inside the job, any time an ActiveRecord model needs to access the database, it uses a database connection from the pool of available connections shared by all ActiveRecord models in this process. The connection pool lets a thread take a connection or blocks until a free connection is available.

If you have less connections available in your ActiveRecord database connection pool than running Sidekiq jobs/threads, jobs will be blocked waiting for a connection and possibly timeout (after ~ 5 seconds) and fail.

This is why it's important that you have as many available database connections as threads in your sidekiq worker process.

Unicorn is a single-threaded, multi-process server - so you shouldn't need more than one connection for each Unicorn back-end worker process.

However, the database can only handle so many connections (depending on OS, hardware, and configuration limits) so you need to make sure that you are distributing your database connections where they are needed and not exceeding your maximum.

For example, if your database is limited to 1000 connections, you could only run 20 sidekiq processes with 50 threads each and nothing else.