Designing a multi-client tcp server to process data

Question

I am attempting to rewrite my current project to include more features and stability, and need some help designing it. Here is the jist of it (for linux):

• TCP_SERVER receives connection (auth packet)
• TCP_SERVER starts a new (thread/fork) to handle the new client
• TCP_SERVER will be receiving many packets from client > which will be added to a circular buffer
• A separate thread will be created for that client to process those packets and build a list of objects
• Another thread should be created to send parts of the list of objects to another client

The reason to separate all the processing into threads is because server will be getting many packets and the processing wont be able to keep up (which needs to be quick, as its time sensitive) (im not sure if tcp will drop packets if the internal buffer gets too large?), and another thread to send to another client to keep the processing fast as possible.

So for each new connection, 3 threads should be created. 1 to receive packets, 1 to process them, and 1 to send the processed data to another client (which is technically the same person/ip just on a different device)

And i need help designing this, as how to structure this, what to use (forks/threads), what libraries to use.

Answer 1

Trying to do this yourself is going to cause you a world of pain. Focus on your actual application, and leverage an existing socket handling framework. For example, you said:

for each new connection, 3 threads should be created

That statement says the following: 1. You haven't done this before, at scale, and haven't realized the impact all these threads will have. 2. You've never benchmarked thread creation or synchronous operations. 3. The number of things that can go wrong with this approach is pretty overwhelming.

Give some serious thought to using an existing library that does most of this for you. Getting the scaffolding right around this can literally take years, and you're better off focusing on your code rather than all the random plumbing.

The Boost C++ libraries seem to have a nice Async C++ socket handling infrastructure. Combine this with some of the existing C++ thread pools and you could likely have a highly performant solution up fairly quickly.

I would also question your use of C++ for this. Java and C# both do highly scalable socket servers pretty well, and some of the higher level language tooling (Spring, Guarva, etc) can be very, very valuable. If you ever want to secure this, via TLS or another mechanism, you'll also probably find this much easier in Java or C# than in C++.

Some of the major things you'll care about: 1. True Async I/O will be a huge perf and scalability win. Try really hard to do this. The boost asio library looks pretty nice. 2. Focus on your features and stability, rather than building a new socket handling platform. 3. Threads are expensive, avoid creating them. Thread pools are your friend.

Answer 2

You plan to create one-or-more threads for every connection your server handles. Threads are not free, they come with a memory and CPU overhead, and when you have many active threads you also begin to have resource contention.

What usage pattern do you anticipate? Do you expect that when you have 8 connections, all 8 network threads will be consuming 100% of a cpu core pushing/pulling packets? Or do you expect them to have a relatively low turn-around?

As you add more threads, you will begin to have to spend more time competing for resources in things like mutexes etc.

A better pattern is to have one or more thread for network io - most os'es have mechanisms for saying "tell me when one or more of these network connections has io" which is an efficiency saving over having lots of individual threads all doing the same thing for just one connection.

Then for actual processing, spin up a pool of worker threads to do actual work, allowing you to minimize the competition for resources. You can monitor work load to determine if you need to spin up more to meet delivery requirements.

You might also want to look into something to implement the network IO infrastructure for you; I've had really good performance results with libevent but then I've only had to deal with very high performance/reliability networking systems.