what is the type of data expected in RSA_verify, can't verify a signature generated from Java

Question

I create a signature on Java using :

String message = "my message";
byte[] data = message.getBytes();
byte[] result;
Signature sig = Signature.getInstance("SHA512withRSA");
sig.initSign(this.privateKey);
sig.update(data);
result = sig.sign();

Then I save result as a Hex String into a text file, and using openssl I try to verify:

string signature//read hex string in the text file 
string message = "my message";
RSA *rsaPkey = NULL;

FILE *pemFile;
fopen_s(&pemFile, publicKeyFile, "r");
rsaPkey = PEM_read_RSA_PUBKEY(pemFile, &rsaPkey, NULL, NULL);
fclose(pemFile);

if(rsaPkey == NULL)
{
    RSA_free(rsaPkey);
    return 0;
}

int type                = NID_sha512;
const char *m           = message.c_str();
unsigned int m_len      = message.length();
int size                = RSA_size(rsaPkey);
unsigned char *sigret   = (unsigned char*) signature.c_str();
unsigned int siglen     = signature.length();

unsigned char digest[SHA512_DIGEST_LENGTH];

SHA512_CTX ctx;
SHA512_Init(&ctx);
SHA512_Update(&ctx, m, m_len);
SHA512_Final(digest, &ctx);

int r = RSA_verify(type, digest, SHA512_DIGEST_LENGTH, sigret, siglen, rsaPkey);

r is always 0, which means the verification has failed. I think it's because it's expecting the message or the signature in a specific format than the hex I got from Java but i don't know what exaclty.

UPDATE

After using the digest as suggested by Pras, I get this message error when I use ERR_get_error : error:04091068:rsa routines:INT_RSA_VERIFY:bad signature

Answer 1

From RSA_verify() man page:

RSA_verify() verifies that the signature sigbuf of size siglen matches a given message digest m of size m_len. type denotes the message digest algorithm that was used to generate the signature. rsa is the signer's public key.

So the second and third arguments are supposed to be message digest and digest length not actual message and message length that was signed

Answer 2

It could be that byte[] data = message.getBytes(); may not give the same sequence of bytes as message.c_str() . I suggest you check it with a debugger.