Sidekiq stuck threads not responding to TTIN signal

Question

We are using sidekiq for our background job processing and we are having problems with sidekiq. While processing jobs sidekiq gets its thread into a stuck state. When I check on sidekiq web interface I don't see any processes running. But I can see a process running like below. ps -aux |grep sidekiq rails 26512 0.2 4.8 1619652 196936 ? Sl 13:10 0:10 sidekiq 3.3.3 railsapp [7 of 20 busy]

I tried to kill with 'kill -TTIN 26512' to get a backtrace but it does not respond to above.

Does anyone know how to debug this ? This keeps happening with our application we often had to kill the process and need to restart the sidekiq process.

Further more we are using Mutex objects in background processing.

Using rails 4.1.2 Oracle ('activerecord-oracle_enhanced-adapter', '1.5.5')

sidekiq yml configuraitons

:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:concurrency: 20
:timeout: 30
:daemon: true
:queues:
  - default
  - task
  - task2
  - task3

Answer 1

Use GDB to get a C-level backtrace:

https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting#frozen-processes

Your backtrace indicates a bug in the Oracle native driver. Thread 9 is holding the GVL while waiting for a native lock. Another thread is holding that native lock, waiting for the GVL. This causes all threads to come to a halt since no one can get the GVL.