PEM_read_RSAPrivateKey returns “Illegal Seek” when decrypting using OpenSSL libs in C

Question

This problem has been driving me nuts! :) I am generating a couple of public/private key pairs, but when i go to load the private key i get an error. I am using C. Note i do use the password "password" on the keys, it is set as a #define but i have tried putting it in manually as a string with no difference. Here is how i'm generating the keys:

FILE *fp;
OpenSSL_add_all_algorithms();
RSA *rsa=NULL;
unsigned char seed[KEYSIZE];
int i;

//Seed PRNG
srand(time(NULL));
RAND_bytes(seed, KEYSIZE - 1);

//Generate a key
if ((rsa=RSA_generate_key(KEYSIZE,65537,NULL,NULL)) == NULL) ERR_get_error();

//Write the public key
fp = fopen(pubkey,"w");
if (!PEM_write_RSA_PUBKEY(fp, rsa)) {
    printf("Error writing public key\n"); exit(1);
}
fclose(fp);

//Write the private key
fp = fopen(privkey,"w");
if (!PEM_write_RSAPrivateKey(fp, rsa, EVP_des_ede3_cbc(), NULL, 0, NULL, PASSWORD))
{
   printf("Error writing private key\n"); exit(1);
}
fclose(fp);

This completes successfully and i end up with the files in the appropriate places, and they look good. I've also successfully encrypted/decrypted on the command line using these files, so i know they work!

Here is how i'm loading the private key:

static void decrypt(int locale) {
FILE *key; 
RSA *rsa;

key = fopen(PRIVATEKEY, "r");
if (key == NULL) perror("Error reading private key");

rsa = PEM_read_RSAPrivateKey(key, NULL, NULL, PASSWORD); // THIS BREAKS!
if (rsa == NULL) perror("Private Key not valid");  // I always get this error :(

if (rsa != NULL) RSA_free(rsa);
fclose(key);}

Any suggestions? I went through the source file rsa.c and this is basically exactly how they do it in there, except they use BIOs instead of freads. I don't see why my way shouldn't work though!

Here is the relevant section of strace output:

open("/opt/evoting/keys/priv/mix1.priv", O_RDONLY) = 15
fstat64(15, {st_mode=S_IFREG|0644, st_size=1751, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb773c000
read(15, "-----BEGIN RSA PRIVATE KEY-----\n"..., 4096) = 1751
dup(2)                                  = 16
fcntl64(16, F_GETFL)                    = 0x2 (flags O_RDWR)
fstat64(16, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb773b000
_llseek(16, 0, 0xbfba00d0, SEEK_CUR)    = -1 ESPIPE (Illegal seek)
write(16, "Private Key not valid: Resource "..., 56Private Key not valid: Resource     temporarily unavailable ) = 56
close(16)                               = 0
munmap(0xb773b000, 4096)                = 0
close(15)                               = 0
munmap(0xb773c000, 4096)                = 0

Also, perror produces the error "resource not available". I'm not sure what that's about, but i reset my hard and soft ulimits to be really high and it did nothing. I copied the function into its own file and compiled it and it doesn't produce the error there (it still fails to read the key, though).

Answer 1

Aha! Alright, i just learned a lot about troubleshooting this stuff, and hopefully it will help someone else along the line. Most of my info came from this thread: http://readlist.com/lists/openssl.org/openssl-users/2/10340.html

Basically i needed to not use perror and instead use ERR_print_errors_fp(stderr) . This gave me an actual error message like so:

3077973640:error:0906B072:lib(9):func(107):reason(114):pem_lib.c:530:

I then used the command: openssl errstr 0906B072 and got:

error:0906B072:PEM routines:PEM_get_EVP_CIPHER_INFO:unsupported encryption

Which basically led me back to the creation of my key pair where i specified CBC: EVP_des_ede3_cbc() which is not supported for RSA. So there you go!

Answer 2

I found solution - you just call OpenSSL_add_all_ciphers() function before

OpenSSL_add_all_ciphers() adds all encryption algorithms to the table including password based encryption algorithms.

The cipher and digest lookup functions are used in many parts of the library. If the table is not initialized several functions will misbehave and complain they cannot find algorithms. This includes the PEM, PKCS#12, SSL and S/MIME libraries. This is a common query in the OpenSSL mailing lists.

Calling OpenSSL_add_all_algorithms() links in all algorithms: as a result a statically linked executable can be quite large. If this is important it is possible to just add the required ciphers and digests.

http://www.openssl.org/docs/crypto/x509.html#