RSA encrypt with OpenSSL in Qt/C++ and decrypt in Java: Bad Padding Exception

Question

I am trying out a sample program where I am encrypting a string with RSA public key in C++ Qt Framework (using statically linked OpenSSL C++ library), and decrypting the same ciphertext using javax.crypto library. I am sending this ciphertext through a socket connection using a free port on my PC to the localhost.

The following are the codes:

My Qt/C++ code:

main.cpp:

#include "cipher.h"
#include "assert.h"
#include "string.h"
#include <QApplication>
#include <QTcpSocket>

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
    Cipher *cipher=new Cipher();

    QTcpSocket *qts=new QTcpSocket();
    qts->connectToHost("localhost",11111);

    QByteArray message, enc_key,enc_message;

    message="Elephant";
   
    enc_message=cipher->encryptRSA(cipher->getPublicKey("publickey.pem"),message);

    qDebug()<<message;
    qDebug()<<enc_message;

    if(qts->waitForConnected(300)){
        qts->write(QString::fromStdString(enc_message.toStdString()).toUtf8().constData());
        qts->write("\n");
        qts->flush();
    }

    return a.exec();
}

For the encryptRSA function I used the example from VoidRealms' tutorial. Here is the GitHub link: https://github.com/voidrealms/Qt-154

From the above link I used cipher.h and cipher.cpp without any changes.

Java code:

package servertest;

import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
import java.security.InvalidKeyException;
import java.security.KeyFactory;
import java.security.NoSuchAlgorithmException;
import java.security.PrivateKey;
import java.security.spec.PKCS8EncodedKeySpec;
import java.util.Base64;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.crypto.Cipher;
import javax.crypto.NoSuchPaddingException;
import org.apache.commons.io.FileUtils;

public class ServerTest {
    static int port=11111;
    static ServerSocket ss;
    static Socket s;
    
    public static void main(String[] args) {
        System.out.println("Server Started!");
        try {
            ss = new ServerSocket(port);
            s=ss.accept();
            
            InputStreamReader isr = new InputStreamReader(new BufferedInputStream(s.getInputStream()));
            BufferedReader br = new BufferedReader(isr);
            
            String str=br.readLine();
            
            System.out.println("Received: "+str);
            byte[] encrypted = str.getBytes("UTF-8");
            Cipher cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
            PrivateKey privateKey = loadPrivateKey();
            cipher.init(Cipher.DECRYPT_MODE, privateKey);
            byte[] decrypted = cipher.doFinal(encrypted); 
            
            System.out.println("Decrypted: "+new String(decrypted));
            
        } catch (IOException ex) {
            Logger.getLogger(ServerTest.class.getName()).log(Level.SEVERE, null, ex);
        } catch (InvalidKeyException ex) {
            Logger.getLogger(ServerTest.class.getName()).log(Level.SEVERE, null, ex);
        } catch (NoSuchAlgorithmException ex) {
            Logger.getLogger(ServerTest.class.getName()).log(Level.SEVERE, null, ex);
        } catch (NoSuchPaddingException ex) {
            Logger.getLogger(ServerTest.class.getName()).log(Level.SEVERE, null, ex);
        } catch (Exception ex) {
            Logger.getLogger(ServerTest.class.getName()).log(Level.SEVERE, null, ex);
        }
    }
    
    public static PrivateKey loadPrivateKey() throws Exception {
        String privateKeyPEM = FileUtils.readFileToString(new File("privatekey-pkcs8.pem"), StandardCharsets.UTF_8);

        // strip off header, footer, newlines, whitespaces
        privateKeyPEM = privateKeyPEM
                .replace("-----BEGIN PRIVATE KEY-----", "")
                .replace("-----END PRIVATE KEY-----", "")
                .replace("-----BEGIN RSA PRIVATE KEY-----", "")
                .replace("-----END RSA PRIVATE KEY-----", "")
                .replaceAll("\\s", "");
        
        //System.out.println(privateKeyPEM);
        
        // decode to get the binary DER representation
        byte[] privateKeyDER = Base64.getDecoder().decode(privateKeyPEM.getBytes("UTF-8"));

        KeyFactory keyFactory = KeyFactory.getInstance("RSA");
        PrivateKey privateKey = keyFactory.generatePrivate(new PKCS8EncodedKeySpec(privateKeyDER));
        return privateKey;
    }
    
}

Also, I generated these keys using standard OpenSSL commands. I tried using different keys with different bit lengths but I get the same error. I have converted the privatekey.pem to PKCS8 ( privatekey-pkcs8.pem ). For generating and using the keys, I followed the link below:

https://adangel.org/2016/08/29/openssl-rsa-java/

THE PROBLEM:

I am getting javax.crypto.BadPaddingException: Decryption error in Java. What am I doing wrong?

THINGS I ALREADY TRIED

• I am new to encryption and I don't know if I should encode this ciphertext to something like base64 or hex, when I tried this, Java complains that the ciphertext is longer than maximum bits allowed.
• At the qts->write() stage, I tried converting between several datatypes and formats including const char*, char[], QByteArray, toUtf8().toconstData(), std::string, converting to QString using both QString::fromUtf8() and QString::fromLocal8bit(). Should I try Utf16 and Latin1?

Please help me with this one.

Answer 1

Okay, I SOLVED this problem, within 20 minutes after posting, by converting the ciphertext to Base64 and passing it through the socket and, of course, on the receiving side, I need to decode it back to bytes.

Previously when I tried this, I forgot to decode the Base64 text back to bytes. Sorry if I wasted somebody's time.