wc_RsaSSL_Verify returns BAD_FUNC_ARG, and I can't tell why

Question

I am trying to RSA public key decrypt a signed file using wolfcrypt - yes, I may or may not be abusing the "sign/verify" power of RSA to encrypt a separate AES key using the private key and decrypt using the public key.

Unfortunately, I am stuck at wc_RsaSSL_Verify() - for the life of me I can't figure out why it is returning BAD_FUNC_ARG - I figured an error like that should be immediately visible to somebody else so I'm deciding to call upon the collective powers of StackOverflow.

As far as I can tell, I'm giving the function what it's asking for - an input buffer, an output buffer, the size of each, and a pointer to the RsaKey struct. Here is a code snippet from the function in question:

bool VerifyWorker::GetAESKey()
{
  bool result = true;
  uint8_t en_aes_file_buff[VerifyWorkerLocal::RSA_KEY_SIZE];
  uint8_t de_aes_file_buff[VerifyWorkerLocal::RSA_KEY_SIZE];

  uint8_t* aes_iv_ptr = NULL;

  // keyfile filestream
  std::fstream aes_file;

  // rsa_key must be initialized
  if(rsa_key == NULL)
  {
     result = false;
  }

  // Open the key file and read it into a local buffer, then decrypt it and use it to initialize the
  // aes struct
  if(result)
  {
     aes_file.open(this->aes_key_file, std::ios_base::in | std::ios_base::binary);

     if(aes_file.fail())
     {
        // Unable to open file - perror?
        perror("GetAESKey");
        result = false;
     }
     else
     {
        aes_file.read(reinterpret_cast<char*>(en_aes_file_buff), VerifyWorkerLocal::RSA_KEY_SIZE + 1);
        if(!aes_file.eof())
        {
           // we didn't have enough space to read the whole signature!
           std::cerr << "aes_file read failed! " << aes_file.rdstate() << std::endl;
           result = false;
        }
     }
  }

  // "Unsign" the aes key file with RSA verify, and load the aes struct with the result
  if(result)
  {
     int wc_ret = 0;
     wc_ret = wc_RsaSSL_Verify(const_cast<const byte*>(en_aes_file_buff),
                               VerifyWorkerLocal::RSA_KEY_SIZE, reinterpret_cast<byte*>(&de_aes_file_buff),
                               VerifyWorkerLocal::RSA_KEY_SIZE, rsa_key);

The rsa_key is a private member initialized (successfully, using wc_PublicKeyDecode()) in a separate function with a public key DER file. I generated both the public and private key using OpenSSL - which should properly pad my AES key and iv file using PKCS#1 v1.5 b default.

I should also mention that I am using wolfssl version 3.9.8. Thanks!

Answer 1

The issue, I found, was that the file that I had signed with my RSA key was not signed correctly. When I signed the file using OpenSSL, my cli invocation was

openssl rsautl -in keyfile -out keyfile -inkey private.pem -sign

Apparently, openssl does not like you to specify the same file for -in and -out. When I changed it to something like

openssl rsautl -in keyfile -out keyfile_signed -inkey private.pem -sign

I was actually able to verify the file using wc_RsaSSL_Verify.

So, like most stupid late-night, last hour software problems, I was looking in the wrong place entirely. I was a bit thrown off by the BAD_FUNC_ARG being returned and thought that it had to do explicitly with the format of the function arguments, not necessarily their content. Hopefully this answer is useful for somebody else, too.

Answer 2

It sounds like you are trying to use RSA_Sign to perform an "Encrypt" of an AES key. Then I assume you are sending to a remote partner or computer who will then run an RSA_Verify operation to decrypt the AES key do I understand the scenario correctly?

If so I apologize it did not show up if you searched on how to do this initially but we actually have an example of doing exactly that here:

https://github.com/wolfSSL/wolfssl-examples/tree/master/signature/encryption-through-signing

That example includes two separate applications. The first app, "rsa-private-encrypt-app.c", will sign (encrypt) the "fake Aes Key" and output the result to a file. The second app, "rsa-public-decrypt-app.c", then opens the file that was output and does a verify (decrypt) on the data contained in the file to recover the original "fake Aes Key".

I may or may not be abusing the "sign/verify" power of RSA to encrypt a separate AES key using the private key and decrypt using the public key.

No not at all, that is a valid use of RSA sign/verify ASSUMING you are working with fixed-length inputs such as an AES key.

That's why we created the example! We actually had a user ask a very similar question on our forums awhile back which led to us making the example.

One thing to make note of though on the issues you encountered with openssl and wolfssl is actually talked about in the README:

https://github.com/wolfSSL/wolfssl-examples/blob/master/signature/encryption-through-signing/README.md

... Keep in mind this is not a TRUE RSA ENCRYPT and will likely not inter-op with other libraries that offer a RSA_PRIVATE_ENCRYPT type API.

This is a true SIGN operation.

If you have any other questions feel free to post them here (and add the wolfssl tag of course) or you can also send us an email anytime at support@wolfssl.com

Disclaimer: I work for wolfSSL Inc.