Encrypt a SecretKey with RSA in Java

Question

I'm working on a client-server secure protocol where I need to use RSA in Java to encrypt a SecretKey for HMAC digests because the key has to be sent to the server. The encryption has two stages; first, I need to encrypt the symmetric key with a public asymmetric key, then, that encrypted message is encrypted with a private asymmetric key.

For this purpose I generate the SecretKey as:

public SecretKey generate(){
KeyGenerator generator = KeyGenerator.getInstance("HMACSHA256");
k = generator.generateKey();
return k;
}

Later, I use this code to encrypt any byte array with a public key:

public byte[] encryptPublic(PublicKey key, byte[] array){
Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.ENCRYPT_MODE, key);
byte[] encrypted = cipher.doFinal(array);
return encrypted;
}

The code for encryption with a private key is the same but using a private key.

For the RSA encryption I'm using 1024 bit long asymmetric keys so I have two main questions:

1. How can I turn my SecretKey to a byte array in order to encrypt it with RSA and a public key?
2. As the public key encryption produces a byte array with 128 bytes, how can I encrypt that message again with a private key if the key is 1024 bits long and can only encrypt a 117 byte long message?

Answer 1

1. How can I turn my SecretKey to a byte array in order to encrypt it with RSA and a public key?

That's called wrapping:

public static byte[] wrapKey(PublicKey pubKey, SecretKey symKey)
        throws InvalidKeyException, IllegalBlockSizeException {
    try {
        final Cipher cipher = Cipher
                .getInstance("RSA/ECB/OAEPWithSHA1AndMGF1Padding");
        cipher.init(Cipher.WRAP_MODE, pubKey);
        final byte[] wrapped = cipher.wrap(symKey);
        return wrapped;
    } catch (NoSuchAlgorithmException | NoSuchPaddingException e) {
        throw new IllegalStateException(
                "Java runtime does not support RSA/ECB/OAEPWithSHA1AndMGF1Padding",
                e);
    }
}

Note that this explicitly doesn't convert to byte[] first. That's because the key might well be within eg a hardware security module. In a HSM the wrapping may be possible, but the conversion to byte[] in local memory would usually not be possible.

---

2. As the public key encryption produces a byte array with 128 bytes, how can I encrypt that message again with a private key if the key is 1024 bits long and can only encrypt a 117 byte long message?

You shouldn't do this and you cannot do this either. The reason that you shouldn't do it because encryption with the private key does not provide confidentiality , as anybody would have access to the public key.

Padding is required to perform secure RSA encryption. The padding overhead (of 11 bytes for PKCS#1 v1.5 style padding) is there prohibiting you to encrypt with the private key.

Note that the entire operation: encryption with a private key isn't even specified in PKCS#1 - it's not a legit operation.

---

Usually the much more secure ephemeral-ephemeral (EC)DH is used to establish keys in transport protocols, using the private key(s) only for authentication. You may want to take a hint from the (draft versions of) TLS 1.3. Or you may just want to use TLS or the handshake portion of it.