how to make c openssl generate the same signed text as JAVA SHA256withRSA/PSS Signature?

Question

I am working on rpc call signing, and to make server accept our API calls, we need to use RSAPrivateKey to sign http mime headers. Server side code is written in JAVA and use "SHA256withRSA/PSS" to verify signature.

My problem is I got different signing hashes from JAVA and c openssl code. So the question is, is it possible for openssl to generate the same signing hash as JAVA?

JAVA code:

public static String getSignedString(PrivateKey privateKey, String text) throws Exception {
    Signature sig = Signature.getInstance("SHA256withRSA/PSS", "BC");
    sig.initSign(privateK);
    sig.update(text.getBytes(StandardCharsets.UTF_8));

    byte[] signed = sig.sign();

    String result = Base64.getEncoder().encodeToString(signed);
    System.out.println("signed : " + result);
    /// signed result works perfect.
}

c code:

bool RSASign(RSA* rsa, const unsigned char* Msg, size_t MsgLen,unsigned char** EncMsg,size_t* MsgLenEnc) {
EVP_MD_CTX* m_RSASignCtx = EVP_MD_CTX_create();
EVP_PKEY* priKey  = EVP_PKEY_new();
EVP_PKEY_CTX *pctx = EVP_PKEY_CTX_new(priKey, NULL);

EVP_PKEY_CTX_set_rsa_padding(pctx, RSA_PKCS1_PSS_PADDING);  
// tried RSA_PKCS1_PADDING w/o success.
EVP_PKEY_assign_RSA(priKey, rsa);
if (EVP_DigestSignInit(m_RSASignCtx,&pctx, EVP_sha256(), NULL,priKey)<=0) {
    return false;
}
if (EVP_DigestSignUpdate(m_RSASignCtx, Msg, MsgLen) <= 0) {
    return false;
}
if (EVP_DigestSignFinal(m_RSASignCtx, NULL, MsgLenEnc) <=0) {
    return false;
}
*EncMsg = (unsigned char*)malloc(*MsgLenEnc);
if (EVP_DigestSignFinal(m_RSASignCtx, *EncMsg, MsgLenEnc) <= 0) {
    return false;
}
// here EncMsg is different from JAVA output, server validation failed. need to make it the same as JAVA output.
}

Answer 1

The "PSS" in "RSA-PSS" stands for probabilistic signature scheme -- it's a randomized algorithm. You aren't supposed to get the same signature every time.