RSA signature verification failing with libtomcrypt

Question

I was using libtomcrypt for a project, in particular hashing and signing with RSA some data. Because of memory requirements I wanted to switch to mbedtls. However I noticed an issue when trying to verify with mbedtls a signature generated by libtomcrypt. There is a padding (PKCS#1 PSS) decoding issue.

Because I didn't know which of these libraries was (maybe) having a bug I used openssl to verify.

I successfully verified a signature generated with mbedtls with openssl and I had a verification failure when verifying one generated form libtomcrypt which makes me think libtomcrypt has a bug.

However because It could be a configuration issue so I created a repo with minimal examples with the hope someone could detect an issue coming from me and not from the library itself.

The test repo is here

Thanks for any help received !

Answer 1

There's one important part missing, you should not be signing the data, but a hash of the data.

Adding this fixes the issue:

diff --git a/test-rsa-sign-verify-libtomcrypt-openssl.c b/test-rsa-sign-verify-
libtomcrypt-openssl.c
index 758994a..25e1312 100644
--- a/test-rsa-sign-verify-libtomcrypt-openssl.c
+++ b/test-rsa-sign-verify-libtomcrypt-openssl.c
@@ -9,6 +9,7 @@
 #include "openssl/err.h"
 #include "openssl/rsa.h"

+#define ERRORe(e) printf("%d: \"%s\"\n", __LINE__, error_to_string(e));
 #define ERROR() printf("Error at line : %d\n", __LINE__);

 char *data = "This is the data which will be hashed and then signed by RSA";
@@ -57,10 +58,18 @@ int main(void)
     printf("* Random number generator registered\n");

     size_t data_length = strlen(data);
+
+    uint8_t hash[32];
+    unsigned long hash_length = sizeof(hash);
+    if ((err = hash_memory(hash_idx, data, data_length, hash, &hash_length)) != CRYPT_OK) {
+       ERRORe(err);
+       return -1;
+    }
+
     unsigned long signature_length = 256;
     unsigned char *signature = calloc(signature_length, sizeof(unsigned char));

-    err = rsa_sign_hash((const unsigned char *) data, data_length,
+    err = rsa_sign_hash((const unsigned char *) hash, hash_length,
                         signature, &signature_length,
                         NULL, prng_idx, hash_idx, 12,
                         &private_key);