OpenSSL RSA_public_encrypt strange behavior

Question

I'm trying to write a simple encryption routine in C using OpenSSL and I've found something strange. I'm not a C guru nor OpenSSL professional. So I might have made a mistake.

The function is as follows

char *rsa_encrypt(char *data)
{
    const char xponent_in_hex[] = "010001";
    const char modulus_in_hex[] = "D0BA16F11907E7B0819705A15264AC29BEE9F1EC5F22642992
    D3E27100B7F212864A624A12FFB6D531712B0B0225AAD0C2E313D077A7DB2A5A33483EEFF41A9D";    

    BIGNUM *xponent = NULL;
    BIGNUM *modulus = NULL;

    BN_hex2bn(&xponent, xponent_in_hex);
    BN_hex2bn(&modulus, modulus_in_hex);

    RSA *rsa = RSA_new();
    rsa->e = xponent;
    rsa->n = modulus;
    rsa->iqmp = NULL;
    rsa->d = NULL;
    rsa->p = NULL;
    rsa->q = NULL;

    char encoded[512] = { 0 };
    RSA_public_encrypt(
    strlen(data), 
    (const unsigned char *)data, 
    (unsigned char *)encoded,
    rsa, 
    RSA_PKCS1_OAEP_PADDING
);

    RSA_free(rsa);

    return (encoded);
}

int _tmain(int argc, _TCHAR* argv[])
{
    printf("%s\n", base64_encode(rsa_encrypt("ABC")));
    printf("%s\n", base64_encode(rsa_encrypt("ABC")));
    printf("%s\n", base64_encode(rsa_encrypt("ABC")));
}

I call that function on same data several times and it generates different value each time it is called. It is apparently wrong because exponent and modulus for created RSA structure are constant and input data is the same in each call.

So why RSA_public_encrypt behaves that way?

How should I generate a public key for RSA encryption based on exponent and modulus ?

And where I've made mistake?

Answer 1

This is actually correct, and you're not making a mistake. Your confusion stems from the RSA_PKCS1_OAEP_PADDING parameter to RSA_public_encrypt .

The RSA encryption process is actually:

1. Take the plaintext (plain) and encode it , producing encoded_plain.
2. Encrypt encoded_plain.

(As you would expect, the decryption process requires you to both decrypt the value, and then decode the message).

The RSA_PKCS1_OAEP_PADDING parameter specifies how the plaintext should be encoded (that OAEP encoding should be used).

A simplified explanation is that OAEP padding uses some random values for the padding, so both xxxxxxxABC and yyyyyyyABC and zzzzzzzABC are all valid encoded_plain values for your plaintext, and those encoded_plain encrypts to a different value. If you perform the corresponding decrypt (and decode, by passing the same RSA_PKCS1_OAEP_PADDING paramater to RSA_private_decrypt ) operation, you should still get "ABC" as an output for each of the ciphertexts, as the padding stripped off all three.

(If you want to be precise, the OAEP encoding scheme is more complicated that, see RFC 3447 section 7.1.1 . But those are probably details you don't care about.)

Answer 2

The scope of encoded ends at the end of the rsa_encrypt function. Your return pointer will point to an invalid area of memory, that might not contain what you expect anymore because somebody else (another thread, for example) wrote over it. The answer explaining the padding is correct.