C++ Public Key Encryption with RSA_public_encrypt

Question

I'm currently struggeling with the public Key Encryption in C++. I am using openssl but the generated string is always broken. My current solution is the following:

string encryptWithPublicKey(const char* pData, int iLen) { // pData: String to encrypt, iLen: length

        char chPublicKey[] = "---Public Key-- ...";

        BIO* bio = BIO_new_mem_buf(chPublicKey, -1);

        RSA* rsa = PEM_read_bio_RSA_PUBKEY(bio, NULL, NULL, NULL);
    

        int nLen = RSA_size(rsa);
        char* pEncode = (char*)malloc(nLen + 1);
        int rc = RSA_public_encrypt(iLen, (unsigned char*)pData, (unsigned char*)pEncode, rsa, RSA_NO_PADDING);

        cout << "From: " << pData << " To: '" << pEncode << "'";

        RSA_free(rsa);
        CRYPTO_cleanup_all_ex_data();
        std::string strName = std::string(pEncode);

        return strName;
}

The output of the encryption is always in that format: "═════════════════════════════════════════════════════════════════²²²²☻"

No idea why the output has that format ^^

Maybe someone has an idea how to use the encryption in a better.

Thanks Kurt

Answer 1

1. Encrypted data is binary. You shouldn't print binary data. You should encode it as base64 or hex before printing.

2. ════════════ are 0xCC bytes, magic fill bytes for uninitialized memory. It seems nothing was writted to the output, which means the call to RSA_public_encrypt failed.

You are not checking the return value rc .

int rc = RSA_public_encrypt(iLen, (unsigned char*)pData, (unsigned char*)pEncode, rsa, RSA_NO_PADDING);

Add some error reporting to see the exact error:

if (rc < 0) {
    char buf[128];
    cerr << "RSA_public_encrypt: " << ERR_error_string(ERR_get_error(), buf) << endl;
}

It will probably print something like RSA_padding_add_none:data too small for key size .

That's because RSA_public_encrypt encrypts exactly RSA_size(rsa) bytes (the size of the RSA key), and is meant primarily for encrypting symmetric keys, which are in turn used for encryption.

I use public/private key for symmetric key transfer

Good. In that case the key should fit and you just need to pad it to RSA_size(rsa) bytes. Or better yet, use some good padding like RSA_PKCS1_OAEP_PADDING instead of RSA_NO_PADDING .

Note that std::string(pEncode) expects a C-string input. You need std::string(pEncode, rc) in order to store binary data inside std::string . Or better yet, use base64 (see EVP_EncodeBlock ).