How to check if a connection is SSL?

Question

I have a C server application that uses OpenSSL and I receive all traffic on the same port. Is there a safe way to check if incoming data is an SSL connection or something else?

Answer 1

The first thing that happens on a TLS connection is the client sending a ClientHello. The ClientHello begins with a byte value 22 ( '\\x16' ) identifying it as a handshake message.

If your application protocol is text-based, then it won't contain any 22 bytes (it's not a printable character) so the first byte is sufficient to distinguish your-protocol-over-TCP from your-protocol-over-TLS-over-TCP.

If your application protocol is not text-based, and it's possible for a non-TLS connection to begin with the client sending a 22 byte, you'll have to dig deeper. The next 2 bytes are the TLS major and minor version numbers; currently you can expect the major version byte to be 3 and the minor version byte to be somewhere in the range 1 to 3 . Lower numbers are possible if you have clients using obsolete, busted versions of SSL, and higher numbers will become possible with future updates to TLS, so you'll have to be flexible.

Hopefully you can just rule out 22 as the first byte of the non-TLS version of your protocol.

You can use recv with MSG_PEEK to inspect the first byte without consuming it, so after you've made the decision it will still be there for the TLS library or your application protocol to read.

Another possible complication: If your application protocol requires the server to speak before the client, you have a problem. After the client connects, it might be waiting for your non-TLS greeting, or it might be sending a ClientHello. This problem can only be solved by a timeout - if the client doesn't send anything within some timeframe, assume it's not going to send a ClientHello and go ahead with the non-TLS version of protocol. This punishes non-TLS clients with a delay in connection startup, but there's no way to avoid that.

Answer 2

If you accept the connections yourself, and hand off to SSL handshaking after you accepted the connection, you need to keep track of it yourself.

A simple solution is to have a structure containing the accepted socket descriptor, and a boolean flag if it's a SSL connection or not, and have a list of those structures.