Encrypt in Java using RSA/ECB/OAEPWithSHA-256AndMGF1Padding, decrypt in c# using OAEPSHA256 Padding

Question

I have a situation where a Java program encrypts text using RSA/ECB/OAEPWithSHA-256AndMGF1Padding.

I need to decrypt it in c#.

Encryption and decryption work fine in Java.

Encryption in Java and decryption in c# with RSA/ECB/OAEPWithSHA-1AndMGF1Padding works absolutely fine.

However, with RSA/ECB/OAEPWithSHA-256AndMGF1Padding encryption in Java and decryption with OaepSHA256 in C# gives me the error : The parameter is incorrect.

Java Code for encrypt:

public static String encrypt(KeyPair keypair, String data) throws NoSuchAlgorithmException, NoSuchPaddingException,
            InvalidKeyException, IllegalBlockSizeException, BadPaddingException, UnsupportedEncodingException {
        Cipher c = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
        //Cipher c = Cipher.getInstance("RSA/ECB/OAEPWithSHA-1AndMGF1Padding");
        c.init(Cipher.ENCRYPT_MODE, keypair.getPublic());
        return Base64.getEncoder().encodeToString(c.doFinal(data.getBytes()));
    }

C # code for decrypt :

public string DecryptRsa(byte[] encryptedBytes, X509Certificate2 x509Certificate2, RSAEncryptionPadding rSAEncryptionPadding)
        {
            var text = string.Empty;

            using (RSACng csp = (RSACng)x509Certificate2.GetRSAPrivateKey())
            {

                byte[] bytesDecrypted = csp.Decrypt(encryptedBytes, rSAEncryptionPadding);
                text = Encoding.UTF8.GetString(bytesDecrypted);
            }
            return text;
        }

What am i doing wrong? Please help.

Answer 1

OAEP uses two digests, the OAEP digest and the MGF1 digest, see RFC8017 .

The SunJCE provider specifies with RSA/ECB/OAEPWithSHA-256AndMGF1Padding the OAEP digest as SHA256, while the MGF1 digest defaults to SHA1, see here . The C# code, on the other hand, specifies with OaepSHA256 both digests as SHA256. Therefore, both codes are incompatible.

The fix is to either explicitly specify the digests in Java with OAEPParameterSpec (which should always be done anyway for this very reason). On the C# side, no fix is possible with on board means, since a separate specification of both digests is not supported. But BouncyCastle can be used which supports this.

---

Fix, Java code side (SHA256 for both digests):

RSAPublicKey publicKey = ...
OAEPParameterSpec oaepParameterSpec = new OAEPParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, PSource.PSpecified.DEFAULT);
Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPPadding");
cipher.init(Cipher.ENCRYPT_MODE, publicKey, oaepParameterSpec);
byte[] ciphertext = cipher.doFinal(plaintext);

Fix, C# code side, using BouncyCastle (SHA256 for OAEP digest, SHA1 for MGF1 digest):

using Org.BouncyCastle.Crypto.Digests;
using Org.BouncyCastle.Crypto.Engines;
using Org.BouncyCastle.Crypto.Encodings;
using Org.BouncyCastle.Crypto.Parameters;
...
RsaKeyParameters privateKey = ...
OaepEncoding oaepEncoding = new OaepEncoding(new RsaEngine(), new Sha256Digest(), new Sha1Digest(), null);
oaepEncoding.Init(false, privateKey);
byte[] decrypted = oaepEncoding.ProcessBlock(ciphertext, 0, ciphertext.Length);