Sidekiq工作忙于使用postgres和rails 5

Question

I got some "randomly" jobs stuck in busy state in sidekiq for hours (even days). I am not able to reproduce it in my local dev environment, but in production DB (azure db service) got stuck.

When I send TTIN to SQ I get this:

2017-07-24T09:13:47.000Z [1]: Thread TID-qf9sw processor                                                                                           [1673/1844]
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/postgresql_adapter.rb:598:in `async_
exec'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/postgresql_adapter.rb:598:in `block 
in exec_no_cache'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in
 log'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activesupport-5.0.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/postgresql_adapter.rb:598:in `exec_n
o_cache'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/postgresql_adapter.rb:587:in `execut
e_and_clear'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:10
3:in `exec_query'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:373:
in `select'
2017-07-24T09:13:47.000Z [1]: /app/vendor/bundle/ruby/2.3.0/gems/activerecord-5.0.4/lib/active_record/connection_adapters/abstract/database_statements.rb:41:i
n `select_all'

I can see in PG that there lot of idle connections, but they get used over time, so seems like normal pool behavior.

Running same process "by hand" runs instantly so seem not to be a code problem. I am also using #with_connection but didn't help.

I am using 2 docker containers, one running rails server (puma) other running just sidekiq.

I tried tweaking the variables and reaper setting in database.yml but didn't solve the problem.

Any clue will be very welcome.

Answer 1

What are the jobs doing? Are they creating deadlocks at a db level or other somehow? Putting some kind of debug log in the job will help you understand where within the job it's getting stuck. Sidekiq has timeout and uniqueness settings (may require licensing) which may help manage the issue but not fix the root.

Answer 2

因此，这是Azure服务端的“未定义”问题，感谢您的尝试！