Getting different encryption results - RSA - Java and Openssl

Question

When I encrypt a string of text with Java and Openssl - I am unable to decrypt the cipher (using Javascript Crypto Subtle) got from Openssl, however Java cipher is decrypted (with Crypto Subtle Web API JS) succesfully. Could you please look into and advise where is the mistake?

1. Here are RSA Public and Private key examples in Base64 generated by JS Crypto Sublte API:

Public key (RSA-OAEP, monulus 2048, hash SHA-256):

MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8JhmO71HZ1vb8uxKhw6MM+ZvtTmc9Tw6AFpJVgXKiTjebj1SPdnxhdhJ5Bj15RN0rGNACXhAyUdn7zsp66/tmjNuC91L+9PvQBjDbXLsx7XUV9nIGJ3aYO5/qruVNXwyemf7wlwZVDF7oeZ8JUfjABTg7a7xui2WdXDHgvhTQdvQ9TS0NkX9xWAiDSn/HWfoEBC7TLeKeVjHsT7g1JnskGxfVhFrLfxQCxYZle4ebXP7dCPsff4WNNCxzBQtHHt8iEoZ0SVKBzS5zhdLHIdAIW4ELdnYsM7iTlWZO+kfWnlV2i13lAJobhxOAqwsg4OqkTsrx0KtWfZH40bNtFzx8wIDAQAB

Private Key (the same parameteres):

   MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDwmGY7vUdnW9vy7EqHDowz5m+1OZz1PDoAWklWBcqJON5uPVI92fGF2EnkGPXlE3SsY0AJeEDJR2fvOynrr+2aM24L3Uv70+9AGMNtcuzHtdRX2cgYndpg7n+qu5U1fDJ6Z/vCXBlUMXuh5nwlR+MAFODtrvG6LZZ1cMeC+FNB29D1NLQ2Rf3FYCINKf8dZ+gQELtMt4p5WMexPuDUmeyQbF9WEWst/FALFhmV7h5tc/t0I+x9/hY00LHMFC0ce3yIShnRJUoHNLnOF0sch0AhbgQt2diwzuJOVZk76R9aeVXaLXeUAmhuHE4CrCyDg6qROyvHQq1Z9kfjRs20XPHzAgMBAAECggEAPtCjPGyeFFu00LclfB5tt20t9CQ/GP3o7MelxvpLF0mMNT74VmKs/rNqE037ARxzxUBCa1aEn6hvd9O8DadIgw5zaFCWMoDyQYtVlqE/NaMA9hDLf7XS2qTaGyLPgX/UFAZLAkMWb9ddfncVKYybtR0+Xn/i56dYVYAk7spTvmkai5Q4Li5eEJqrRNse5fg/wiwDda/cWlEvlvfziAjU51gS/Y9ItUT8Z1g8A2NYfqgFVgBVb0qrtbP76TWPbeLZr1FXTMDB0MO+zuC+1lq48+CYqSJqi9a4GlW2gi0wkaQe3VIOsHipv82kSh20LczXiSKqF58F8m4whWqpa02rrQKBgQD6tbuFiY8fVXi44m1mFO7KDIVXFjuMca0+qXisVaUhG4q42fnlHN0kyuH2IFXNxHD+gE4MnvAUkXDkG1znaz8vM2tfQn6vSpU7OytwOt6UEMWm+112083nGMUy1Me/6rAJbBWfTAl0KXY9/Y/SuEL8u+W6Eas+eASUIe1DooI4xwKBgQD1rAfUXjuRH6WFnbITdG8B3WVZhRCx3YuQVE1UnHR8N6qfx6qKIricvKMSvtWAj0UiIbSA8DENyfMiyAbGpZ0cILxtdXZOtZFW9hfiOeLptnYKTJXOiGoju1prxpUNUKu/1kd7e0clxN75mooge8yFFA6il43wYhhwO2si+FYJdQKBgQDL+1TpX3StX9NrSf8MkXd/uRQ8OQCWUl9MnoJqZPyHpWsG34Ms4IElUFTs9n4Zfv0YdLgMGLzpXzRkw8ahG2c7NjDkPqvoX1xv5sJ++8bg3YyTQe1XoxjiMAsyQmGLSp2T7PbitvDyLFHiOg3surL2AsL00y9rEidXhwsOfohJPQKBgQCqhw4sQHjShIgNlmfMj06amcZG/FGZVPISbiH7cFp++tjp7duX5QAGc/4x/dsPUOOpDIJR2egC7UJiyzvA2aaTprmEtTs46VmIZmwvsQSsO+X1wjFeWlxqjxr1orNFudBt6dxWfzzkn6Iy2i203Jobac+61r5EtKLIDMaSUJTQHQKBgQDaNTfgakpyZBSp/Ydzu76qDKpzGuDqrLBnR/lui4DrSyUoOjNwaGlkU/B6USdaC/BGO7lI08HZU7FzutvXb1mKS+nUI7rKvSTN9NKUJF4MaBv9q40RViKkF9vbq2f1wMs2KUNc0ex2ZL+5BOBICJNHyVulKSxY54Z55L/H4FZmWw==

2. Here is Java Code which receives Public key as Base64 string and create PublicKey instance from ite. Then I am encrypting a string of text with this key and encode the cipher to Base64 string.

    String stringToEncrypt = "This is encrypted message;..". byte[] publicKeyBytes = Base64.getDecoder().decode("MIIBIjANBgkqhki.;.;"), Cipher cipher = Cipher,getInstance("RSA/ECB/OAEPPadding"), OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-256". "MGF1". new MGF1ParameterSpec("SHA-256");PSource.PSpecified;DEFAULT). KeyFactory keyFactory = KeyFactory;getInstance("RSA"). PublicKey rsaPublicKey = keyFactory.generatePublic(new X509EncodedKeySpec(publicKeyBytes)), cipher,init(Cipher;ENCRYPT_MODE. rsaPublicKey.oaepParams). String cipherBytes = Base64.getEncoder();encodeToString(cipher.doFinal(stringToEncrypt.getBytes()));

   The cipherBytes Base64 string then succsefully decrypted by JS Crypto Subtle code below:

3. First keypair generation code:

const keyPair = await window.crypto.subtle.generateKey(
            {
                name: "RSA-OAEP",
                modulusLength: 2048,
                publicExponent: new Uint8Array([1, 0, 1]),
                hash: "SHA-256"
            },
            true,
            ["encrypt", "decrypt"]
        )

Import Private key =

    return window.crypto.subtle
          .importKey("pkcs8", base64StringToArrayBuffer(privateKeyInPem), {
              name: "RSA-OAEP",
              hash: "SHA-256",
          }, true, ["decrypt"]).then(key=>{
              return key;
          })

And then decrypt cipher

        const dec = new TextDecoder();

        function str2ab(str: any) {
            const buf = new ArrayBuffer(str.length);
            const bufView = new Uint8Array(buf);
            for (let i = 0, strLen = str.length; i < strLen; i++) {
                bufView[i] = str.charCodeAt(i);
            }

            return buf;
        }

        console.log('cipher: '+cipher)
        return window.crypto.subtle.decrypt({
                name: "RSA-OAEP",
            },
            privateKey,
            str2ab(window.atob(cipher))
        ).then(result=>{
            console.log('DECRYPTED MSG:'+dec.decode(result));
        })

But the problem is when I try to decrypt the cipher from openssl - it doesn't work. Here is the Openssl command which I use:

OpenSSL> rsautl -encrypt -pubin -keyform DER -inkey publicKey.der -oaep -in input.txt -out out.bin

I am using the same Public key, just converted from Base64 string to DER format (converted Base64 string key to DER with Java - 1. Decode base64 string -> byte[]->write to file each byte)

And finally last Openssl operation to convert binary data from out.bin to Base64 string

OpenSSL> enc -A -base64 -in out.bin -out base64.txt

Answer 1

rsautl unchangeably uses SHA1 for the OAEP digests, see here . Since you apply SHA256 in the WebCrypto code, you must usepkeyutl instead, which allows the specification of the digests:

openssl pkeyutl -in plaintext.txt -out ciphertext.bin -encrypt -keyform DER -pubin -inkey spki.der -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha256 

A ciphertext generated in this way can be decrypted using the WebCrypto code:

 (async () => { // your private key var privateKeyInPem = "MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDwmGY7vUdnW9vy7EqHDowz5m+1OZz1PDoAWklWBcqJON5uPVI92fGF2EnkGPXlE3SsY0AJeEDJR2fvOynrr+2aM24L3Uv70+9AGMNtcuzHtdRX2cgYndpg7n+qu5U1fDJ6Z/vCXBlUMXuh5nwlR+MAFODtrvG6LZZ1cMeC+FNB29D1NLQ2Rf3FYCINKf8dZ+gQELtMt4p5WMexPuDUmeyQbF9WEWst/FALFhmV7h5tc/t0I+x9/hY00LHMFC0ce3yIShnRJUoHNLnOF0sch0AhbgQt2diwzuJOVZk76R9aeVXaLXeUAmhuHE4CrCyDg6qROyvHQq1Z9kfjRs20XPHzAgMBAAECggEAPtCjPGyeFFu00LclfB5tt20t9CQ/GP3o7MelxvpLF0mMNT74VmKs/rNqE037ARxzxUBCa1aEn6hvd9O8DadIgw5zaFCWMoDyQYtVlqE/NaMA9hDLf7XS2qTaGyLPgX/UFAZLAkMWb9ddfncVKYybtR0+Xn/i56dYVYAk7spTvmkai5Q4Li5eEJqrRNse5fg/wiwDda/cWlEvlvfziAjU51gS/Y9ItUT8Z1g8A2NYfqgFVgBVb0qrtbP76TWPbeLZr1FXTMDB0MO+zuC+1lq48+CYqSJqi9a4GlW2gi0wkaQe3VIOsHipv82kSh20LczXiSKqF58F8m4whWqpa02rrQKBgQD6tbuFiY8fVXi44m1mFO7KDIVXFjuMca0+qXisVaUhG4q42fnlHN0kyuH2IFXNxHD+gE4MnvAUkXDkG1znaz8vM2tfQn6vSpU7OytwOt6UEMWm+112083nGMUy1Me/6rAJbBWfTAl0KXY9/Y/SuEL8u+W6Eas+eASUIe1DooI4xwKBgQD1rAfUXjuRH6WFnbITdG8B3WVZhRCx3YuQVE1UnHR8N6qfx6qKIricvKMSvtWAj0UiIbSA8DENyfMiyAbGpZ0cILxtdXZOtZFW9hfiOeLptnYKTJXOiGoju1prxpUNUKu/1kd7e0clxN75mooge8yFFA6il43wYhhwO2si+FYJdQKBgQDL+1TpX3StX9NrSf8MkXd/uRQ8OQCWUl9MnoJqZPyHpWsG34Ms4IElUFTs9n4Zfv0YdLgMGLzpXzRkw8ahG2c7NjDkPqvoX1xv5sJ++8bg3YyTQe1XoxjiMAsyQmGLSp2T7PbitvDyLFHiOg3surL2AsL00y9rEidXhwsOfohJPQKBgQCqhw4sQHjShIgNlmfMj06amcZG/FGZVPISbiH7cFp++tjp7duX5QAGc/4x/dsPUOOpDIJR2egC7UJiyzvA2aaTprmEtTs46VmIZmwvsQSsO+X1wjFeWlxqjxr1orNFudBt6dxWfzzkn6Iy2i203Jobac+61r5EtKLIDMaSUJTQHQKBgQDaNTfgakpyZBSp/Ydzu76qDKpzGuDqrLBnR/lui4DrSyUoOjNwaGlkU/B6USdaC/BGO7lI08HZU7FzutvXb1mKS+nUI7rKvSTN9NKUJF4MaBv9q40RViKkF9vbq2f1wMs2KUNc0ex2ZL+5BOBICJNHyVulKSxY54Z55L/H4FZmWw=="; var privateKey = await window.crypto.subtle.importKey("pkcs8", base64StringToArrayBuffer(privateKeyInPem), {name: "RSA-OAEP", hash: "SHA-256"}, true, ["decrypt"]); console.log(privateKey); // Base64 encoded ciphertext from openssl pkeyutl var cipher = "V2QWzwoOHPAlKhGqv0fDSv+lSPytBW4tTxVVJgneyvfIqTDvllhbZJzUAszdAC0IEow+YgbIWHyIBCw9YVS+EDZ3jbuIU97nx5NxAimiUFKvUmHE8p3oP6AP/etJhGQGC+fMiTbhmGn5FQhMnH/2lVei3yJypXWbgI6ONRmncYalq73q7VGelFUSubuPWQA3bKzuIOSorpQFy9sGIMDvW+YOMLrClVmUujVrEXrdsIbvzSb6hooKHbwjOaAmN4XRw0sr+YaF3n2PwazpLSvJmuugF26GxhmJAMmNViUvvsN+ycpJZdyRKNehQGqmahpC0XXihZ9dsHEH7vIDDmPAZQ=="; console.log('cipher: '+cipher) // successfull decryption var decrypted = await window.crypto.subtle.decrypt({name: "RSA-OAEP"}, privateKey, str2ab(window.atob(cipher))); const dec = new TextDecoder(); console.log('DECRYPTED MSG:' + dec.decode(decrypted)); })(); function base64StringToArrayBuffer(base64String){ return Uint8Array.from(window.atob(base64String), c => c.charCodeAt(0)); } function str2ab(str) { const buf = new ArrayBuffer(str.length); const bufView = new Uint8Array(buf); for (let i = 0, strLen = str.length; i < strLen; i++) { bufView[i] = str.charCodeAt(i); } return buf; }

---

For completeness: In the context of OAEP there are two digests to specify, the content digest and the MGF1 digest. pkeyutl allows to set both independently. If both digests are identical (which is usually the case), it is sufficient to set only the content digest via rsa_oaep_md , ie the MGF1 digest implicitly has the same value as long as no other digest is explicitly set via rsa_mgf1_md .