RSA encryption using public key. Data size based on key

Question

I'm using the OpenSSL RSA API to encrypt data with a server's public key.

uint32_t rsasize = RSA_size(keypair); // -- RSA key size == 256

uint32_t dl = 255
uint8_t *d; // points to 255 bytes of data

unsigned char *encrypt = (unsigned char*)malloc(rsasize);

auto num = RSA_public_encrypt(dl, d, encrypt, keypair, RSA_NO_PADDING);
if ( num == -1 )
{
    auto err = malloc(130);
    ERR_load_crypto_strings();
    ERR_error_string(ERR_get_error(), (char*)err);
    printf("Error encrypting message: %s\n", err);
    return nullptr;
}

I'm using RSA_NO_PADDING so RSA should crypt 255 bytes with 256 byte public key easily. But I'm receiving this:

Error encrypting message: error:0406B07A:rsa routines:RSA_padding_add_none:data too small for key size

I changed dl(data_lenght) to 256 (just +1) and I got this:

Error encrypting message: error:04068084:rsa routines:RSA_EAY_PUBLIC_ENCRYPT:data too large for modulus

I know that RSA can encode 255 bytes with 256-key. What's the problem?

Answer 1

You are padding at the wrong end. If you have 255 bytes of data and you want to pad it out to 256 you should add a zero byte at the high-order end . In other words, you should insert a 0 just before d[0].