Hiding encryption key AES-256 in JAVA

Question

As we now, a .jar file, we can open it and see the code and classes with any decompiler. Now suppose the following situation:

I developed a Client-Server application under JAVA, in both Client and Server, i used AES-256 for encrypting the data sent over internet. So, i give the "Client" to my friend. Now.. The question is... can he hack my Server knowing how are the packets received in the Client, treated, and sent back? I mean... encrypting the data when you have a revealed code, it's.. in vain, doesn't it?

The question is... what can i do to have the best security possible in the Server? does encryption work in this case ?

Thanks !!!

Answer 1

Don't use encryption just to hide how your client-server protocol works. If you want your server to be secure, make it secure even when the client is in full control of what goes on.

As you've mentioned, it's easy to control the client: not just via decompilation, but also by running the client in a debugger (which can modify the contents of any object, modify control flow, etc.).

My points, then, are:

1. It's important to identify where the potential attack vectors are in your server, and address those. For example, if you want to be able to send a cookie to a client, which you want them to send back unaltered, you don't need encryption for that; just use HMAC.

2. If you actually really want to encrypt traffic between client and server, don't use straight AES (especially because it sounds like you're using a fixed key, which is insecure). You want to use a protocol designed for wire encryption, like TLS (SSL). Yes, really use TLS (including acquiring a certificate from a certificate authority); don't take shortcuts.

   The nice thing about TLS is that there are no shared secrets (such as keys) that you have to embed into your jar. So there is nothing you have to worry about hiding.

Answer 2

If by him hacking your server you mean him meaning figuring out the protocol and contents of the packet that goes in, and then sending funky packets to make your server do unexpected things at unexpected times, the solution seems to be not as much encryption but strong validation. That is, your server should expect only a limited set of values in the incoming packets at each step of your protocol, and if some of the incoming packets do not conform to this/attempt to do something unexpected, they should be discarded.

Answer 3

Your encryption is not going to allow you to hide the protocol/API/etc. that the client and server use to communicate with each other. By encrypting the data flowing between the client and server, you are preventing someone that is observing this traffic as it passes across the wire (presumably the internet) from snooping on the data.

One issue to keep in mind is that AES requires a password that is know by both the client and the server. If your password is hardcoded in the jar, then anyone that has the jar can snoop on any traffic between your server and anyone's client that connects to it. So, you need a secure way for the client and server to agree on the password to use for a given connection.

I suggest using an SSL-based technology for encrypting the data going on the wire (eg, maybe you can simply use HTTPS as your protocol). Or perhaps use public key encryption.

It is hard to make a specific recommendation, given the lack of information about the communication pattern between client and server.