How to distinguish a DTLS packet from TCP, UDP packet

Question

From RFC 5764 :

               +----------------+
               | 127 < B < 192 -+--> forward to RTP
               |                |
   packet -->  |  19 < B < 64  -+--> forward to DTLS
               |                |
               |       B < 2   -+--> forward to STUN
               +----------------+

Where B is the first byte of the packet. So for DTLS packet identification we do the following after casting the data to unsigned char*,

if(packet[0] > 19 && packet[0] < 64)

This works only for RTP, DTLS and STUN packets. But fails for UDP, TCP etc. If I send a UDP packet which has the first Byte between 20 and 63 then the above condition considers it as a DTLS packet. From google's source code DTLS packet identification is done like this,

static bool IsDtlsPacket(const char* data, size_t len) {
  const uint8* u = reinterpret_cast<const uint8*>(data);
  return (len >= 13 && (u[0] > 19 && u[0] < 64));
}

But this doesn't identify DTLS packets properly as well. In wireshark I can see that it can correctly distinguish DTLS packets from UDP/TCP packets. How do I do it?

Answer 1

RFC 5764 care only about RTP, STUN and DTLS and not about other data. The algorithm to distinguish between these formats is sufficient in this context. If you want to distinguish DTLS from general UDP/TCP you have to know that

• DTLS is a datagram protocol and therefore usually not used on top of TCP at all. On TCP you usually find TLS instead.
• Since DTLS is usually used on top of UDP, any DTLS packet is a valid UDP packet, but not the other way.

To distinguish DTLS packets from other UDP packets you should look at the packet format as described in RC 6347 . The quality of your detection depends on the inspection technology used, that is if you restrict yourself to match only some bytes (like version or type fields) , if you get some more context (does the length makes sense) or if you even have detection engines for other protocols too so that you can be more sure that it is DTLS and not something else.