C++/Linux: how do you write a thread-safe library that uses sockets?

Question

I want to write a library in C++ under Linux that will help an application to use a certain protocol (FastCGI, actually). The library will listen to a socket (either TCP or Unix), receive requests, forward them to user code, and send responses generated by said user code.

There will be many connections on the socket and each connection will carry many requests (possibly simultaneously - there is an interleaving mechanism). The user code (which uses the library) will most likely be multithreaded in order to process several requests in parallel.

I'd like my library to be robust and make as little assumptions/requirements about the user code as possible, including the type of multithreading used. As I understand, the clone() function in Linux can fork a process in dozens of different manners - with or without shared memory, shared file handles, etc. The decision of HOW to implement multithreading should be left to the user.

And this confuses me, because the library code can suddenly find itself fork() 'ed, and multiple copies of the code can be suddenly reading from the same socket and handling the same request. Even worse - the parent process might terminate, leaving only child processes, which in turn spawn more child processes, perhaps even in different process namespaces - it's a mess.

What are the Linux facilities that help to coordinate all the copies of the same code which need to access the same external resource (a socket)? What is the standard way of implementing such thread-safe libraries? Must I choose a threading model myself and impose that upon the consumers of my library?

Answer 1

Don't use directly clone (reserve clone to implementors of threading libraries like pthread ). Don't use a lot of fork -s (probably none). Go using pthread -s.

You could look at the design of the libonion library. It is small, implements HTTP server protocol, so is quite similar to your goals.

^{libonion gives the users various modes for creating or not threads for requests.}

You could have options similar to libonion -s about creating, or not, a new thread for each FastCGI request.

You might perhaps want to use some event looping library like libevent or libev (around a poll(2) -ing loop).

And read good books, notably Advanced Linux Programming , and some tutorial on Pthread -s before starting coding.

Also, study the source code of several free software libraries similar to your goals.

Answer 2

At the risk of seemingly going off at a tangent I'd recommend implementing fastcgi on a single thread per processor basis.

Reasons:

1. More robust.
2. Avoids context-switching overhead associated with multi-threading and protects you from issues like concurrency deadlocks.
3. Avoids process fork() costs (although quite light it all adds up) and protects you from dealing with potential child zombie processes amongst other headaches.

This would leave you with the choice of implementing the fastcgi interface using :

1. Non-blocking synchronous I/O ( Reactor design pattern): block until a read or write request comes in, pass request to the appropriate handler and then block until the next request comes in.
2. Asynchronous I/O ( Proactor design pattern): pass read and write requests to the operating system where the O/S supports I/O completion events. On Windows that would be IO completion ports and on Linux something like epoll() .