Prevent people from reverse-engineering my network protocol

Question

I know I cant prevent people from reverse-engineering my protocol but I'd like to take a security-through-obscurity approach to make it as hard as possible.

I have a server/client system that communicates through the network with http style packets.

Example:

Header
Attribute: Value
Attribute2: Other Value

Payload

I would like to make it as hard as possible for anything other than my client to access the network. Pushing problems with them decompiling my assemblies aside - what would be some good things I could do to this network spec that would make it VERY DIFFICULT to understand and make another implementation without the source?

I was thinking some kind of strange hashing approach or some kind of encryption algorithm that would be difficult.

EDIT I'm not trying to protect my assemblies or source-code. I'm trying to prevent someone from, for example, watching my protocol with WireShark or similar and then making their own implementation based on that information.

Answer 1

All right, three cases:

• Users can't access server code and can't access client code: Easiest way is to use a pregenerated shared secret stored in the binary, and aes encrypt/decrypt.

• Users can access client or server code but not both: Use a public/private key method. You can encrypt using the public key but the private one is needed to decrypt.

• Users can access both client and server code: You're screwed.

If you want to improve security, this static key should only be used during session initiation, to generate a new shared secret, which is then used for communication.

Edit: actually, a more easy and safe solution is to use ssl and certificates (it's a mantra that you shouldn't implement your own encryption) Each certificate comes with a secret private key. As long as users don't have access to that you're safe if you verify that the peer has the exact correct certificate.

Answer 2

For having reversed a few network protocols (from MMOs), I can tell you that you will never protect your protocol for very long, I'm sorry.

The best you can do is:

• Obfuscate it using a custom algorithm (because it takes longer to reverse than a known one). Using a known encryption scheme offers no protection whatsoever.
• Add noise. Try to be very, very confusing. Add random values that make no sense whatsoever. Try to use a dynamic layout for packets. Move fields. Send useless packets. Just like if it were garbage.
• Version your protocol, so that two consecutive version are incompatible. That can be hard to do, but it obliges the reverser to re-do the work for every subsequent version.

But these are just ways to slow down attackers. It's certainly not going to stop them.

Answer 3

There are three solutions:

1. TLS.
2. SSLv3.
3. Whatever you cook up.

1 and 2 already work.

Answer 4

I am also in the process of writing a network protocol in C#. I have made use of encryption to secure the protocol. Here is the outlay of how I did it, you might find this useful.

• As soon as the client connects the server requests a random UUID from the client, the client encrypts this UUID with a password know to both the server and client.
• All packets sent thereafter by the server or client will encrypted using that UUID as a key.

Regarding "I was thinking some kind of strange hashing approach", hashing is usually only used for data verification. ie to ensure it was not modified en-route to you.

Hope this helps.