Decrypt char* using PKCS7_decrypt() in openssl

Question

I want to encrypt and decrypt using PKCS7_encrypt() and PKCS7_decrypt() functions in openSSL. I have used the example in openSSL Demo. What I want to do is to encrypt a message in the format of char* and decrypt using that char* . I don't want to read and write into file. Here is the code to encrypt, which perfectly works and has no problem:

in = BIO_new_file("encr.txt", "r");
if (!in)
    return 0;

/* encrypt content */
p7 = PKCS7_encrypt(recips, in, EVP_des_ede3_cbc(), flags);
if (!p7)
    return 0;

char* chEnc = new char[1000];

BIO* memorybio = BIO_new(BIO_s_mem());
BIO* base64bio = BIO_new(BIO_f_base64());
BIO* outbio = BIO_push(base64bio, memorybio);

/* Copy PKCS#7 */
long ll = i2d_PKCS7_bio(outbio, p7);
BIO_flush(outbio);
BIO_set_flags(memorybio, BIO_FLAGS_MEM_RDONLY);
BIO_get_mem_data(memorybio, &chEnc);
cout << chEnc << "\n";

Now, when I want to do the reverse and decrypt the char* chEnc, I did as below:

BIO *in = NULL, *out = NULL, *tbio = NULL;
X509 *rcert = NULL;
EVP_PKEY *rkey = NULL;
PKCS7 *p7 = NULL;
int ret = 1;

OpenSSL_add_all_algorithms();
ERR_load_crypto_strings();

/* Read in recipient certificate and private key */
tbio = BIO_new_file("signer.pem", "r");

if (!tbio)
    return 0;

rcert = PEM_read_bio_X509(tbio, NULL, 0, NULL);
BIO_reset(tbio);
rkey = PEM_read_bio_PrivateKey(tbio, NULL, 0, NULL);
if (!rcert || !rkey)
    return 0;

BIO* memorybio = BIO_new(BIO_s_mem());
int iLength = BIO_puts(memorybio, chEnc);

BIO* base64bio = BIO_new(BIO_f_base64());
BIO* inbio = BIO_push(base64bio, memorybio);

/* Copy PKCS#7 */
BIO_flush(inbio);
BIO_set_flags(inbio, BIO_FLAGS_MEM_RDONLY);
p7 = d2i_PKCS7_bio(inbio, &p7);
if (!PKCS7_decrypt(p7, rkey, rcert, out, 0))
    return 0;

ret = 0;

if (ret) {
    fprintf(stderr, "Error Signing Data\n");
    ERR_print_errors_fp(stderr);
}

if (p7)
    PKCS7_free(p7);
if (rcert)
    X509_free(rcert);
if (rkey)
    EVP_PKEY_free(rkey);

if (in)
    BIO_free(in);
if (out)
    BIO_free(out);
if (tbio)
    BIO_free(tbio);

return ret;

The problem is that the PKCS7_decrypt does not work and it does not decrypt into out variable. After the line if (!PKCS7_decrypt(p7, rkey, rcert, out, 0)) return 0; , it returns from the function. Is the procedure to decrypt correct? Shall I use other APIs of openSSL to convert or something?

Looking forward to your suggestions and comments.

Thanks

Answer 1

Are you on your recipient list? Because you need to be the recipient to decrypt the message. Make sure you use the correct key and certificates.

Answer 2

PKCS7_decrypt(p7, rkey, rcert, out, 0) is failing because the BIO "out" that you are passing to the routine is a null pointer.

If you do not want to write the output into a file, try passing a memory BIO:

BIO *out = BIO_new(BIO_s_mem());

Then you can access the decrypted data directly from out, and once you are done with it you can destroy it.