Can't unblock/“wake up” thread with pthread_kill & sigwait

Question

I'm working on a C/C++ networking project and am having difficulties synchronizing/signaling my threads. Here is what I am trying to accomplish:

1. Poll a bunch of sockets using the poll function
2. If any sockets are ready from the POLLIN event then send a signal to a reader thread and a writer thread to "wake up"

I have a class called MessageHandler that sets the signals mask and spawns the reader and writer threads. Inside them I then wait on the signal(s) that ought to wake them up.

The problem is that I am testing all this functionality by sending a signal to a thread yet it never wakes up.

Here is the problem code with further explanation. Note I just have highlighted how it works with the reader thread as the writer thread is essentially the same.

// Called once if allowedSignalsMask == 0 in constructor
// STATIC
void MessageHandler::setAllowedSignalsMask() {
     allowedSignalsMask = (sigset_t*)std::malloc(sizeof(sigset_t));
     sigemptyset(allowedSignalsMask);
     sigaddset(allowedSignalsMask, SIGCONT);
}

// STATIC
sigset_t *MessageHandler::allowedSignalsMask = 0;

// STATIC
void* MessageHandler::run(void *arg) {
    // Apply the signals mask to any new threads created after this point
    pthread_sigmask(SIG_BLOCK, allowedSignalsMask, 0);

    MessageHandler *mh = (MessageHandler*)arg;
    pthread_create(&(mh->readerThread), 0, &runReaderThread, arg);

    sleep(1); // Just sleep for testing purposes let reader thread execute first
    pthread_kill(mh->readerThread, SIGCONT);
    sleep(1); // Just sleep for testing to let reader thread print without the process terminating

    return 0;
}

// STATIC
void* MessageHandler::runReaderThread(void *arg) {
    int signo;
    for (;;) {
            sigwait(allowedSignalsMask, &signo);

            fprintf(stdout, "Reader thread signaled\n");
    }

    return 0;
}

I took out all the error handling I had in the code to condense it but do know for a fact that the thread starts properly and gets to the sigwait call.

The error may be obvious (its not a syntax error - the above code is condensed from compilable code and I might of screwed it up while editing it) but I just can't seem to find/see it since I have spent far to much time on this problem and confused myself.

Let me explain what I think I am doing and if it makes sense.

1. Upon creating an object of type MessageHandler it will set allowedSignalsMask to the set of the one signal (for the time being) that I am interested in using to wake up my threads.
2. I add the signal to the blocked signals of the current thread with pthread_sigmask. All further threads created after this point ought to have the same signal mask now.
3. I then create the reader thread with pthread_create where arg is a pointer to an object of type MessageHandler.
4. I call sleep as a cheap way to ensure that my readerThread executes all the way to sigwait()
5. I send the signal SIGCONT to the readerThread as I am interested in sigwait to wake up/unblock once receiving it.
6. Again I call sleep as a cheap way to ensure that my readerThread can execute all the way after it woke up/unblocked from sigwait()

Other helpful notes that may be useful but I don't think affect the problem:

• MessageHandler is constructed and then a different thread is created given the function pointer that points to run. This thread will be responsible for creating the reader and writer threads, polling the sockets with the poll function, and then possibly sending signals to both the reader and writer threads.

I know its a long post but do appreciate you reading it and any help you can offer. If I wasn't clear enough or you feel like I didn't provide enough information please let me know and I will correct the post.

Thanks again.

Answer 1

POSIX threads have condition variables for a reason; use them. You're not supposed to need signal hackery to accomplish basic synchronization tasks when programming with threads.

Here is a good pthread tutorial with information on using condition variables:

https://computing.llnl.gov/tutorials/pthreads/

Or, if you're more comfortable with semaphores, you could use POSIX semaphores ( sem_init , sem_post , and sem_wait ) instead. But once you figure out why the condition variable and mutex pairing makes sense, I think you'll find condition variables are a much more convenient primitive.

Also, note that your current approach incurs several syscalls (user-space/kernel-space transitions) per synchronization. With a good pthreads implementation, using condition variables should drop that to at most one syscall, and possibly none at all if your threads keep up with each other well enough that the waited-for event occurs while they're still spinning in user-space.

Answer 2

This pattern seems a bit odd, and most likely error prone. The pthread library is rich in synchronization methods, the one most likely to serve your need being in the pthread_cond_* family. These methods handle condition variables , which implement the Wait and Signal approach.

Answer 3

Use SIGUSR1 instead of SIGCONT. SIGCONT doesn't work. Maybe a signal expert knows why.

By the way, we use this pattern because condition variables and mutexes are too slow for our particular application. We need to sleep and wake individual threads very rapidly.

R. points out there is extra overhead due to additional kernel space calls. Perhaps if you sleep > N threads, then a single condition variable would beat out multiple sigwaits and pthread_kills. In our application, we only want to wake one thread when work arrives. You have to have a condition variable and mutex for each thread to do this otherwise you get the stampede. In a test where we slept and woke N threads M times, signals beat mutexes and condition variables by a factor of 5 (it could have been a factor of 40 but I cant remember anymore....argh). We didn't test Futexes which can wake 1 thread at a time and specifically are coded to limit trips to kernel space. I suspect futexes would be faster than mutexes.