Trouble loading RSA public key from file

Question

I've been using openssl library in my C++ project recently, and I'm facing an issue that i can't fix by myself.

I'm actually trying to load an RSA public key stored in a file and encrypt 64 bytes. My code works when it uses a public key generated using the function RSA_generate_key, but when i'm using my own public key, it won't work anymore for some reason.

I've suspected key format from pkcs1 pkcs8, tried both PEM_read_RSAPublicKey and PEM_read_RSA_PUBKEY, still not working for some reason...

This is my public key :

-----BEGIN RSA PUBLIC KEY-----
MEYCQQDE91cW7INdIyVon5H/he2b/DIR25wWT0GFLiZOVp0oAgCAVKDvRZ5+Pqu4
f65XbnNUNNHRJLMLEb1t4JgUhgFVAgER
-----END RSA PUBLIC KEY-----

The key from RSA_generate_key function from Openssl library, which is working :

-----BEGIN RSA PUBLIC KEY-----
MEYCQQDsg/4Qm153/Pr8JRruC0SnVvTrWg/lIPheezIpkwVeWjNz9lMDXNUjdK8v
QgfNUCRJYbnxYIeruAdwTzS/bDXbAgER
-----END RSA PUBLIC KEY-----

And here's my code :

RSA.h :

#include <iostream>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <string>

#ifndef RSA_ALGORITHM_H
#define RSA_ALGORITHM_H

#define KEY_LENGTH       512
#define PUBLIC_EXPONENT  17
#define PUBLIC_KEY_PEM   1
#define PRIVATE_KEY_PEM  0

#define LOG(x)               \
        std::cout << x << std::endl;   \

 /*
  * @brief   create_RSA function creates public key and private key file
  *
  */
RSA* create_RSA(RSA* keypair, int pem_type, char* file_name);

/*
 * @brief   public_ecrypt function encrypts data.
 * @return  If It is fail, return -1
 */
int public_encrypt(int flen, unsigned char* from, unsigned char* to, RSA* key, int padding);

/*
 * @brief   private_decrypt function decrypt data.
 * @return  If It is fail, return -1
 */
int private_decrypt(int flen, unsigned char* from, unsigned char* to, RSA* key, int padding);

/*
 * @brief   create_ecrypted_file function creates .bin file. It contains encrypted data.
 */
void create_encrypted_file(char* encrypted, RSA* key_pair);

#endif //RSA_ALGORITHM_H

RSA.cpp :

#include "RSA.h"

#include <iostream>
#include <string.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <sstream>
#include <iomanip>

int public_encrypt(int flen, unsigned char* from, unsigned char* to, RSA* key, int padding) {
    int result = RSA_public_encrypt(flen, from, to, key, padding);
    return result;
}

void create_encrypted_file(char* encrypted, RSA* key_pair) {

    FILE* encrypted_file = fopen("encrypted_file.bin", "w");
    fwrite(encrypted, sizeof(*encrypted), RSA_size(key_pair), encrypted_file);
    fclose(encrypted_file);
}

RSA* createRSA(int pem_type, char* file_name) {

    RSA* rsa = NULL;
    FILE* fp = NULL;

    if (pem_type == PUBLIC_KEY_PEM) {

        fp = fopen(file_name, "rb");
        PEM_read_RSAPublicKey(fp, &rsa, NULL, NULL);
        fclose(fp);

    }
    else if (pem_type == PRIVATE_KEY_PEM) {

        fp = fopen(file_name, "rb");
        PEM_read_RSAPrivateKey(fp, &rsa, NULL, NULL);
        fclose(fp);

    }

    return rsa;
}

int main() {

    LOG("RSA has been started.");

    char public_key_pem[11] = "public_key";

    RSA* public_key = createRSA(PUBLIC_KEY_PEM, public_key_pem);
    LOG("Public key pem file has been created.");;

    char message[KEY_LENGTH] = "\xc8\xcd\x21\x74\xb9\x84\x33\xb9\x30\x94\xb3\x60\x26\xde\x12\x5a\x7f\x5e\xd8\x5e\xc2\x7e\xe6\xbb\x9e\x99\x6c\xb3\xb9\x38\xe9\xc6\x23\x8c\xc6\x5d\x36\x15\xfb\x63\x5f\x6f\x08\x0f\x6d\xda\x06\x31\x59\x28\xbc\xae\x4c\xcf\x80\x2f\x96\x80\x54\x7d\xb5\x7b\x82\x83";
    char* encrypt = NULL;


    LOG(KEY_LENGTH);
    LOG(PUBLIC_EXPONENT);

    encrypt = (char*)malloc(RSA_size(public_key));
    int encrypt_length = public_encrypt(RSA_size(public_key), (unsigned char*)message, (unsigned char*)encrypt, public_key, RSA_NO_PADDING);
    if (encrypt_length == -1) {
        LOG("An error occurred in public_encrypt() method");
    }
    LOG("Data has been encrypted.");

    create_encrypted_file(encrypt, public_key);
    LOG("Encrypted file has been created.");

    free(public_key);
    free(encrypt);
    LOG("RSA has been finished.");

    return 0;
}

I've seen many posts and haven't found any fix, even though that this one was extremely similar to my issue

. Load public key to create rsa object for public encryption

Answer 1

The error is actually mathematically related. Neither the key nor the the code is causing the issue. If you change the key or the data to encrypt, it'll work fine for some reason. So i guess something is happening during RSA calculation, but no smart enough to figure out what.

Answer 2

Both keys are public RSA keys with a size of 512 bits and an exponent of 17, specified in PKCS1-PEM format.

The message m , the modulus n_fail of the not working key and the modulus n_ok of the working key are:

m =      0xc8cd2174b98433b93094b36026de125a7f5ed85ec27ee6bb9e996cb3b938e9c6238cc65d3615fb635f6f080f6dda06315928bcae4ccf802f9680547db57b8283
n_fail = 0xc4f75716ec835d2325689f91ff85ed9bfc3211db9c164f41852e264e569d2802008054a0ef459e7e3eabb87fae576e735434d1d124b30b11bd6de09814860155
n_ok =   0xec83fe109b5e77fcfafc251aee0b44a756f4eb5a0fe520f85e7b322993055e5a3373f653035cd52374af2f4207cd50244961b9f16087abb807704f34bf6c35db

A comparison shows that

n_fail < m < n_ok

For RSA the condition m < n must apply. This condition is violated for n_fail , which is the cause of the issue. This also means that the corresponding key itself is not invalid. It can be used to encrypt messages that do not violate m < n . For the posted message, however, its modulus is too small.

Regarding security: Nowadays the key should have a size of 2048 bits and one of the paddings specified in RFC8017 ( RSAES-PKCS1-v1_5 or RSAES-OAEP ) should be used.