检测C Pthread中的等待线程

Question

I have a thread pool with about 100 threads. During testing, when I introduce some anomalous conditions, the overall process becomes very slow. Once I make the things normal, the process becomes fast again. Therefore, all threads are running.

I want to detect which threads get slow in particular. For this, I want to write another thread whose responsibility will be to keep an eye on other threads, and report periodically which of them is waiting for a resource to get released. Is there a way (in Pthread) I can find which threads are waiting for some resource to get released, ie which threads are "hung" -- if it is a right term to use?

System: C, Pthread, Linux

PS: Please mention in comments if you need any other details.

Answer 1

I'm probably really old-fashioned, but I say just instrument your code and measure it yourself. For example, add something like the following code (temporarily) to your program, and do a search-and-replace to change all your program's pthread_mutex_lock() calls to instrumented_pthread_mutex_lock().

Then run your program with stdout redirected to a file. Afterwards, you can look in the file and see which threads were waiting a long time for which mutexes.

(Note that the printf() calls will change the timing of your program somewhat, but for this purpose I don't think it will matter much)

#include <stdio.h>
#include <unistd.h>
#include <sys/times.h>

static unsigned long long GetCurrentClockTimeMicroseconds()
{
   static clock_t _ticksPerSecond = 0;
   if (_ticksPerSecond <= 0) _ticksPerSecond = sysconf(_SC_CLK_TCK);

   struct tms junk; clock_t newTicks = (clock_t) times(&junk);
   return ((((unsigned long long)newTicks)*(1000000))/_ticksPerSecond);
}

int instrumented_pthread_mutex_lock(pthread_mutex_t * mtx)
{
   unsigned long long beforeTime = GetCurrentClockTimeMicroseconds();
   int ret = pthread_mutex_lock(mtx);
   unsigned long long afterTime = GetCurrentClockTimeMicroseconds();

   unsigned long long elapsedTime = (afterTime-beforeTime);
   if (elapsedTime > 1000)  // or whatever threshold you like; I'm using 1 millisecond here
   {
      printf("Thread %li took %llu microseconds to acquire mutex %p\n", (long int) pthread_self(), elapsedTime, mtx);
   }
   return ret;
}