Deadlock with threads calling down on kernel semaphore through sysfs

Question

Originating from this question (and my solution ), I have come to realize there is a possible deadlock, but I can't understand why and how I can avoid it.

In short, there is a semaphore in kernel space that kernel modules (they are really applications running in kernel space) could take, but user space applications would also need to take the same semaphore for protecting a globally shared memory.

I have done this by exposing a sysfs file which given the correct character, would down or up the semaphore in kernel space. The user space applications would just keep this file open and write the appropriate character for the lock to take place.

Here's a sample kernel module for demonstration:

#include <linux/module.h>
#include <linux/semaphore.h>
#include <linux/sysfs.h>
#include <linux/kobject.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Shahbaz Youssefi");
MODULE_DESCRIPTION("Test module");

static struct kobject *_kobj = NULL;
static struct semaphore sem;

static ssize_t _lock_op(struct kobject *kobj, struct kobj_attribute *attr, const char *buf, size_t count)
{
    switch (buf[0])
    {
    case '0':
        printk("down (%u)\n", sem.count);
        if (down_interruptible(&sem))
            printk("error: sem wait interrupted\n");
        break;
    case '1':
        printk("up (%u)\n", sem.count);
        up(&sem);
        break;
    default:
        printk("error: invalid request %d\n", buf[0]);
    }
    return count;
}

static struct kobj_attribute _lock_attr = __ATTR(test, 0222, NULL, _lock_op);

static int __init _main_init(void)
{
    sema_init(&sem, 1);

    _kobj = kobject_create_and_add("test", NULL);
    if (!_kobj)
    {
        printk("error: failed to create /sys directory for test\n");
        return -ENOMEM;
    }
    if (sysfs_create_file(_kobj, &_lock_attr.attr))
        printk("error: could not create /sys file\n");

    printk("loaded\n");
    return 0;
}

static void __exit _main_exit(void)
{
    if (_kobj)
        kobject_put(_kobj);
    _kobj = NULL;

    printk("unloaded\n");
}

module_init(_main_init);
module_exit(_main_exit);

This works great in general. The user-space applications can write '0' or '1' to the sysfs file and they achieve mutual exclusion without a problem.

However, there is one case where this locks up the process, and that is when multiple threads of the same process try to acquire the lock.

Essentially, something like this:

       Thread 1               Thread 2

       write '0'
      system call
       _lock_op
   down_interruptible
   return from syscall
                              write '0'
                             system call
                              _lock_op
                           down_interruptible (blocked)

*go on to release the lock*

                            *return from syscall*
                         *go on to release the lock*

The problem is that in such a case, where the second down happens while the first one still hasn't released the lock, instead of just the second thread getting blocked, the whole process gets blocked . That is, the steps marked with * don't happen.

Here's a user-space application that can trigger this when the above kernel module is inserted:

#include <stdio.h>
#include <pthread.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

static int fid;
static volatile sig_atomic_t interrupted = 0;

static void sig_handler(int signum)
{
    interrupted = 1;
}

static void *func(void *arg)
{
    while (!interrupted)
    {
        write(fid, "0", 1);
        write(fid, "1", 1);
        usleep(1000);
    }

    return NULL;
}

int main(void)
{
    pthread_t tid;

    struct sigaction sa = {
        .sa_handler = sig_handler,
    };
    sigemptyset(&sa.sa_mask);
    sigaction(SIGSEGV, &sa, NULL);
    sigaction(SIGINT, &sa, NULL);
    sigaction(SIGHUP, &sa, NULL);
    sigaction(SIGTERM, &sa, NULL);
    sigaction(SIGQUIT, &sa, NULL);
    sigaction(SIGUSR1, &sa, NULL);
    sigaction(SIGUSR2, &sa, NULL);

    fid = open("/sys/test/test", O_WRONLY);
    if (fid < 0)
        return EXIT_FAILURE;

    pthread_create(&tid, NULL, func, NULL);

    while (!interrupted)
    {
        write(fid, "0", 1);
        write(fid, "1", 1);
        usleep(793);
    }

    pthread_join(tid, NULL);

    close(fid);

    return 0;
}

Note: do echo 1 > /sys/test/test to unblock yourself ;)

My question is, why does Linux block the whole process on down rather than just the calling thread? And what can I do about it?

---

Note: tested on x86, kernel 3.8 patched with RTAI. I will try with a newer an vanilla kernel later just to be sure, but I suspect it is unrelated to RTAI.

Answer 1

I have actually figured out a way to go around this problem, but I still think there should be a proper explanation and a solution.

My workaround is the following:

Take a pthread mutex in the application. On each thread, instead of:

write(fid, "0", 1);
/* access */
write(fid, "1", 1);

do

pthread_mutex_lock(&mutex);
write(fid, "0", 1);
/* access */
write(fid, "1", 1);
pthread_mutex_unlock(&mutex);

This makes all accesses by the process to the sysfs file mutually exclusive. The sysfs file makes sure the accesses are mutually exclusive among processes and kernel modules.