RSA_Private_Decrypt fails randomly

Question

I am not a crypto expert, but I wrote encrypt/decrypt methods. I don't know what I am doing wrong here. The encryptKey method returns the variable text length every time for the same string, which is may be due to RSA_PKCS1_PADDING which I am using, but for same string of text, the decryptKey function fails if the encrypted text length is below some threshold.

//----------------------Encryption -----------------------
int Asymmetric::encryptKey(unsigned char *data, int data_len, unsigned char *encrypted)
{
    RSA * rsa = createRSA(1); // Loads the public key
    int result = RSA_public_encrypt(data_len,data,encrypted,rsa,padding);
    return result;
}
//-----------------------Decryption -----------------------
int Asymmetric::decryptKey(unsigned char * enc_data,int data_len, unsigned char *decrypted)
{
    RSA * rsa = createRSA(0); // Loads the private key
    int  result = RSA_private_decrypt(data_len,enc_data,decrypted,rsa,padding);
    return result;
}

How i can overcome this situation?

Edit :
This code does not work

char *myData = "Sample Text";
unsigned char *encrypt = (unsigned char*)malloc(RSA_size(rsaPub));
int result = RSA_public_encrypt(RSA_size(rsaPub)-11 ,(unsigned char*)myData,encrypt,rsaPub,padding);
fprintf(stderr, "\nEnc Size: %d \n", strlen((const char*)encrypt));

Answer 1

If you perform padding, cipher text should have fixed size equal to RSA public key modulus. If you get different size every time, then there is something with your program.

When you perform encryption data must be less than RSA_size(rsa) - 11 and encrypted must point to RSA_size(rsa) bytes of memory.

strlen(const char * str) computes length of null-terminated string. encrypt is not the string and is not null terminated, so using strlen is not valid here. Luckily for you PKCS#1 padding contains zero bytes, so you don't get segmentation fault.

On the other hand myData is a null terminated string. However, you use RSA_size(rsaPub) - 11 as its length, which will result in occasional segmentation faults.