Detect an SslStream (client) connection attempt to an insecure Server (NetworkStream)?

Question

I have a server that is listening for clients with a TcpListener.BeginAcceptTcpClient() method. When a client connects, the EndAcceptTcpClient method gives me the TcpClient object and then I call that client stream's NetworkStream.BeginRead() to listen for data transmitted from the client.

Everything works well until a client (who is expecting the server to support SSL) attempts to connect to the (insecure) server. The data that is retrieved from the NetworkStream.BeginRead() callback when the client attempts to do the SSL handshaking, is gibberish of course, but what is the proper way to detect this handshake after reading the data vs a "normal" message from a client?

Does an SSL handshake have a known header/signature identifier that I could check for by looking at the first few bytes of received messages to distiguish the difference between a normal message and the SSL handshake? From inspection the bytes received, it looks like it may be ASCII 0x16 0x03 0x01 0x00 (aka <SYN><ETX><SOH><NUL> ).

If there is a known header, is there a response I can send to the client to tell it the authentication failed (via the NetworkStream)?

Answer 1

I found some information about the SSL protocol (refs 1 , 2 ) and there is a standard header that I can look for:

From the IBM reference site 2 :

Byte   0       = SSL record type = 22 (SSL3_RT_HANDSHAKE)
Bytes 1-2      = SSL version (major/minor)
Bytes 3-4      = Length of data in the record (excluding the header itself).
Byte   5       = Handshake type
Bytes 6-8      = Length of data to follow in this record
Bytes 9-n      = Command-specific data

So I can look for a message beginning with 0x16 [0x?? 0x??] [0x?? 0x??] 0x01 0x16 [0x?? 0x??] [0x?? 0x??] 0x01 0x16 [0x?? 0x??] [0x?? 0x??] 0x01 (the SSL "Client Hello" message).

For the rejection message, it looks like I can send 0x15 0x?? 0x?? 0x00 0x02 0x02 0x28 0x15 0x?? 0x?? 0x00 0x02 0x02 0x28 0x15 0x?? 0x?? 0x00 0x02 0x02 0x28 which is translated to <NACK> , (2byte version), this is a 2 byte long message, fatal alert, "handshake failed" 3 .

Answer 2

An SSL/TLS client will initiate a handshake by sending a Client Hello message.

In XDR notation, this is:

  struct {
      ProtocolVersion client_version;
      Random random;
      SessionID session_id;
      CipherSuite cipher_suites<2..2^16-2>;
      CompressionMethod compression_methods<1..2^8-1>;
      select (extensions_present) {
          case false:
              struct {};
          case true:
              Extension extensions<0..2^16-1>;
      };
  } ClientHello;

Here, what you're seeing is before that: it's the record layer , for a handshake message ( 0x16 ) with protocol TLS 1.0 ( 0x0301 ). It should be followed by a length and then 0x01 for a "Client Hello" and a version again (possibly different: the highest supported by the client).

From a practical point of view, it's worth looking at actual packets (eg on HTTPS) using Wireshark. There are some sample captures and example on this page (although you won't need to decipher the traffic for what you're trying to do).