OpenSSL RSA encryption with no padding fails to encrypt correctly

Question

We're trying to perform RSA encryption using the "RSA_public_encrypt()" method (openSSL on Symbian), but we're not quite succeeding. The encryption itself succeeds, but the encrypted text (which we try to match to a hash) isn't what it should be (on other platforms, we checked this were we now it is working correctly and we get different values here). We think this is probably due to the input which isn't provided in the correct format to the "RSA_public_encrypt()" method.

The code:

#define RSA_E 0x10001L
void Authenticator::Test(TDesC8& aSignature, TDesC8& aMessage) {   
RSA *rsa;
char *plainkey;
char *cipherkey;
int buffSize;

// create the public key
BIGNUM * bn_mod = BN_new();
BIGNUM * bn_exp = BN_new();

const char* modulus2 = "137...859"; // 309 digits
// Convert to BIGNUM
int len = BN_dec2bn(&bn_mod, modulus2); // with 309 digits -> gives maxSize 128 (OK!)
BN_set_word(bn_exp, RSA_E);

rsa = RSA_new(); // Create a new RSA key
rsa->n = bn_mod; // Assign in the values
rsa->e = bn_exp;
rsa->d = NULL;
rsa->p = NULL;
rsa->q = NULL;

int maxSize = RSA_size(rsa); // Find the length of the cipher text
// maxSize is 128 bytes (1024 bits)

// session key received from server side, what format should this be in ???
plainkey = "105...205"; // 309 digits

cipherkey = (char *)malloc(maxSize);
memset(cipherkey, 0, maxSize);
if (rsa) {
    buffSize = RSA_public_encrypt(maxSize, (unsigned char *) plainkey, (unsigned char *) cipherkey, rsa, RSA_NO_PADDING);

    unsigned long err = ERR_get_error(); 

    free(cipherkey);
    free(plainkey);
}

/* Free */
if (rsa)
    RSA_free(rsa);

}

We have a plain key which is 309 characters, but the maxSize is 128 bytes (RSA_NO_PADDING, so input has to be equal to modulus size), so only the first 128 characters are encrypted (not really sure if this is correct?), which is probably why our encrypted text isn't what it should be. Or is there something else we don't do correctly? What is then the correct way of transforming our plain text so all data of 'plainkey' is encrypted?

We also tried this with a hexadecimal string '9603cab...' which is 256 characters long, but didn't succeed. So if correctly converted into bytes (128) could this be used and if yes, how would that conversion work?

Any help is greatly appreciated, thanks!

Answer 1

Take a look at this file, maybe it helps: https://gist.github.com/850935