Stopping a single-threaded C server using epoll

Question

I am writing a TCP server in C, as an exercise for an assignment I know I'm gonna get next year. I implemented a core that uses a single thread to handle all connections, it uses the epoll facility to do this. The core provides the rest of the program with an API much like C# offers for asynchronous networking; it all works great and after some epoll weirdness -- I suddenly had to use file descriptor duplications to keep it happy, but, luckily, I came to understand why -- I managed to get it bug and leak-free. However, I'm not sure how to stop the server. Currently, I have written cleanup routines that clean up the core data structures, but they are never called. The main epoll_wait loop never ends. How do I stop that loop?

I've thought of the following:

• I use a global flag, guarded by a mutex, that is checked by the core thread every time before calling epoll_wait() . When the flag becomes true, the core thread breaks from its loop and cleans up everything. This works when I use the core thread to set the flag, for example when it exits the core context through a callback (For example, a callback that is invoked when a new connection is made to the listening socket; the core knows nothing about the implementation of these callbacks, and in these callbacks I could set the stop-flag. The mutex wouldn't even be necessary in that case.) However this does not always work when I want an external thread (a GUI event dispatch thread, perhaps? If I want to write a GUI to monitor the server while it is running, that could come in handy) to set that flag. Of course, the core thread will stop the server once it happens to check the flag again, but the problem is that it will not check that flag often if the server is idle (for example when there are no clients and no new connections). Then epoll_wait() will simply remain blocked and I would have to kill the server -- with Ctrl-C or something similar. That is not what I want.

• To solve this, I thought of creating a pipe and add that pipe fd to the epoll context of the server when it starts, so that it will react and return from epoll_wait() whenever some thread writes some (irrelevant) data to that pipe. This could allow some thread to force the core to check its flag and exit. But it feels like a messy solution. Is there a better alternative? If not, will this pipe solution work?

Thanks in advance, as usual, any remarks (if constructive) are welcome!

Answer 1

You stop the loop using one of the usual language constructs, such as break or a condition in a while or for loop.

But I suspect that what you wanted to ask is when to stop the loop. You probably want to do that when your program receives a signal, namely SIGTERM (eg when a service manager such as systemd wants your daemon dead) or SIGINT (eg when you press CTRL+C). If you didn't do anything with signals in the program, your program will stop immediately when either of these two is received, and you don't get a chance to clean up.

By far the easiest way to catch signals in Linux is using the signalfd facility (see man page). This way, you don't need to deal with signal handlers, and their numerous problems such as race conditions, deadlocks and random system calls failing due to a signal. Instead, you get a file descriptor which you register with your epoll instance for readability notifications. When a signal is ready, the file descriptor will be readable and you can read the signal information into a structure. Here you get the chance to break out of the main loop and do cleanup.

But keep in mind that there are very nice libraries (such as libuv) that abstract all these details away and support multiple platforms.

UPDATE

If you want to stop the server from another thread, then you have an instance of a more general problem: communicating information to the event loop. While your idea about sending a signal to yourself will work, it is kind of a hack. There are at least two "proper" solutions: pipe and eventfd , the latter being Linux-specific and specifically designed for this problem. For your use they will both suffice.

For example, with a pipe, you can have the event loop monitor the read end, and have the other thread write a single byte to the write end. But be careful to set the pipe to non-blocking, otherwise the writing thread might block writing to the pipe, and depending on the design, your program could deadlock.

If you need to communicate more information from the thread to the event loop (which you don't for just stopping the server), you can put it somewhere in memory, possibly a queue, with proper locking.