Segfault on openssl RSA_public_encrypt() in C

Question

I have been trying to encrypt a test string using openssl's RSA_public_encrypt but it results to segfault in every way I tried to run it.

I tried checking if the RSA key I am reading from the pem file is valid using BIO and it returned the public key properly with correct key and exponent size.

I had initially tried using PEM_read_bio_RSA_PUBKEY which didn't work out. I am not entirely sure about the difference between that and PEM_read_bio_RSAPublicKey , so if someone can shed some light on this.

Also, before trying out BIO I used the regular the FILE structure and the functions that correspond to that and it kept giving me segfaults and I wasn't able to check if the correct RSA key was loaded either. Not sure if that's related.

#include <stdio.h>
#include <string.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/conf.h>

int main()
{

  RSA *rsa;
  rsa = RSA_new();
  BIO *bp_public = NULL;
  RSA *pubkey;
  bp_public = BIO_new_file("public.pem", "rt");
  pubkey = PEM_read_bio_RSAPublicKey(bp_public, &rsa, NULL, NULL);

  BIO * keybio = BIO_new(BIO_s_mem());
  RSA_print(keybio, rsa, 0);
  char buffer [2048];

  while (BIO_read (keybio, buffer, 2048) > 0)
    {
      printf("%s", buffer);
    }
  BIO_free(bp_public);
  if (pubkey == NULL || rsa == NULL)
    printf("Something went wrong");

  char msg[] = "Hello";
  unsigned char * encrypted = NULL;

  RSA_public_encrypt(5, (unsigned char *)msg, encrypted, rsa, RSA_PKCS1_OAEP_PADDING);

  printf("Here: %s", encrypted);
}

Also, I tried using both pubkey and rsa as the key and none worked.

I am sure I am missing something pretty obvious, but I have spent hours behind it and I am kinda lost in the openssl docs now.

Thanks for the help!

Irrelevant note: The encrypted text returns null if I use RSA_private_encrypt()

Answer 1

From RSA_public_encrypt :

 int RSA_public_encrypt(int flen, unsigned char *from, unsigned char *to, RSA *rsa, int padding); 

RSA_public_encrypt() encrypts the flen bytes at from (usually a session key) using the public key rsa and stores the ciphertext in to . to must point to RSA_size(rsa) bytes of memory.

yet you have to parameter as

unsigned char * encrypted = NULL;

Perhaps you should allocate RSA_size(rsa) bytes of memory for it:

unsigned char *encrypted = malloc(RSA_size(rsa));

Answer 2

Check if you manage various Openssl resources properly. Often times, There would be restrictions on what can be freed. Freeing resources early on might cause a segfault elsewhere in the library.