如何在不破坏连接的情况下发送和接收高达SO_SNDTIMEO和SO_RCVTIMEO的数据？

Question

I am currently planning how to develop a man in the middle network application for TCP server that would transfer data between server and client. It would behave as regular client for server and server for remote client without modifying any data. It will be optionally used to detect and measure how long server or client is not able to receive data that is ready to be received in situation when connection is inactive.

I am planning to use blocking send and recv functions. Before any data transfer I would call a setsockopt function to set SO_SNDTIMEO and SO_RCVTIMEO to about 10 - 20 miliseconds assuming it will force blocking send and recv functions to return early in order to let another active connection data to be routed. Running thread per connection looks too expensive. I would not use async sockets here because I can not find guarantee that they will get complete in a parts of second especially when large data amount is being sent or received. High data delays does not look good. I would use very small buffers here but calling function for each received byte looks overkill.

My next assumption would be that is safe to call send or recv later if it has previously terminated by timeout and data was received less than requested.

But I am confused by contradicting information available at msdn.

send function

https://msdn.microsoft.com/en-us/library/windows/desktop/ms740149%28v=vs.85%29.aspx

If no error occurs, send returns the total number of bytes sent, which can be less than the number requested to be sent in the len parameter.

---

SOL_SOCKET Socket Options

https://msdn.microsoft.com/en-us/library/windows/desktop/ms740532%28v=vs.85%29.aspx

SO_SNDTIMEO - The timeout, in milliseconds, for blocking send calls. The default for this option is zero, which indicates that a send operation will not time out. If a blocking send call times out, the connection is in an indeterminate state and should be closed.

Are my assumptions correct that I can use these functions like this? Maybe there is more effective way to do this?

Thanks for answers

Answer 1

While you MIGHT implement something along the ideas you have given in your question, there are preferable alternatives on all major systems.

Namely:

• kqueue on FreeBSD and family. And on MAC OSX.
• epoll on linux and related types of operating systems.
• IO completion ports on Windows.

Using those technologies allows you to process traffic on multiple sockets without timeout logics and polling in an efficient, reactive manner. They all can be considered successors of the ancient select() function in socket API.

As for the quoted documentation for send() in your question, it is not really confusing or contradicting. Useful network protocols implement a mechanism to create "backpressure" for situations where a sender tries to send more data than a receiver (and/or the transport channel) can accomodate for. So, an application can only provide more data to send() if the network stack has buffer space ready for it.

If, for example an application tries to send 3Kb worth of data and the tcp/ip stack has only room for 800 bytes, send() might succeed and return that it used 800 bytes of the 3k offered bytes.

The basic approach to forwarding the data on a connection is: Do not read from the incoming socket until you know you can send that data to the outgoing socket. If you read greedily (and buffer on application layer), you deprive the communication channel of its backpressure mechanism.

So basically, the "send capability" should drive the receive actions.

As for using timeouts for this "middle man", there are 2 major scenarios:

1. You know the sending behavior of the sender application. Ie if it has some intent on sending any data within your chosen receive timeout at any time. Some applications only send sporadically and any chosen value for a receive timeout could be wrong. Even if it is supposed to send at a specific time interval, your timeouts will cause trouble once someone debugs the sending application.
2. You want the "middle man" to work for unknown applications (which must not use some encryption for middle man to have a chance, of course). There, you cannot pick any "adequate" timeout value because you know nothing about the sending behavior of the involved application(s).

Answer 2

As a previous poster has suggested, I strongly urge you to reconsider the design of your server so that it employs an asynchronous I/O strategy. This may very well require that you spend significant time learning about each operating systems' preferred approach. It will be time well-spent.

For anything other than a toy application, using blocking I/O in the manner that you suggest will not perform well. Even with short timeouts, it sounds to me as though you won't be able to service new connections until you have completed the work for the current connection. You may also find (with short timeouts) that you're burning more CPU time spinning waiting for work to do than actually doing work.

A previous poster wisely suggested taking a look at Windows I/O completion ports. Take a look at this article I wrote in 2007 for Dr. Dobbs. It's not perfect, but I try to do a decent job of explaining how you can design a simple server that uses a small thread pool to handle potentially large numbers of connections:

Windows I/O Completion Ports http://www.drdobbs.com/cpp/multithreaded-asynchronous-io-io-comple/201202921

If you're on Linux/FreeBSD/MacOSX, take a look at libevent:

Libevent http://libevent.org/

Finally, a good, practical book on writing TCP/IP servers and clients is "Practical TCP/IP Sockets in C" by Michael Donahoe and Kenneth Calvert. You could also check out the W. Richard Stevens texts (which cover the topic completely for UNIX.)

In summary, I think you should take some time to learn more about asynchronous socket I/O and the established, best-of-breed approaches for developing servers.

Feel free to private message me if you have questions down the road.