Sidekiq returns “ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)” on AWS RDS

Question

I am creating PDF documents on AWS server with Sidekiq for processing this job on background.

While the process of creating the PDF file, the [Rails] application is pooling database to check out whether the PDF file was created or not (the interval: 2 seconds).

This morning I got this error message on the Sidekiq side:

ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.000 seconds)

I am using Amazon RDS with MySQL on it.

As a temporary solution, I increased the pool parameter from 10 to 30 in database.yml , however I realize this is just a temporary patch.

How to fix it properly?

Thank you

Answer 1

I think that your solution is actually the correct one. ActiveRecord::ConnectionPool is thread based, ie it tries to obtain a separate connection for each thread that wants to work with the database. If there are more threads wanting to access the database then the total size of the connection pool (configured with the pool option in database.yml ), ConnectionPool tries to wait up to 5 seconds by default if a connection from some other thread is freed. After these 5 seconds time out, the ActiveRecord::ConnectionTimeoutError exception is raised.

Now, Sidekiq uses 25 worker threads by default . So, under higher load, it is perfectly possible that there will be up to 25 jobs (threads) trying to access the db at the same time. If your pool was set to 10, the excess workers had to wait for the other ones to complete and probably some thread had to wait too long.

So, either enlarge the size of the connection pool to at least a little higher value then 25 (the number of sidekiq workers), just as you did, or run your sidekiq with less workers by running it like sidekiq -c 5 . Finally, always ensure that you allow enough incoming connections on the MySQL side (by default it's over 100).

Answer 2

Polling generally doesn't scale as a solution.

Without knowing more about the structure of your application I would be tempted to leverage some concurrency constructs from a gem like concurrent-ruby .

Specifically, the creation of a PDF file maps quite closely to the concept of a Future or a Promise

I would look at rearchitecting your worker:

1. Your PDF generation code should be a Promise. It should open a db connection only long enough to write the resulting PDF to the database; not while it is doing pdf generation as well.

2. Your main application code should spin up a PDF generation promise on Sidekiq as usual. Instead of polling the database; this code simply waits for the Promise to complete or fail; if the promise completes succesfully the PDF is in the database, if it fails you have an exception trace etc.

As always, ymmv