Problem with RSA implementation with WinAPI

Question

So I use the CNG framwork from windows in my software(which written exclusively with c). The problem is when I try to implement RSA in my code

The code looks something like this:

    DWORD temp = BCRYPT_SUPPORTED_PAD_OAEP;

    BCryptOpenAlgorithmProvider(PointerToAlgorithmProvider, BCRYPT_RSA_ALGORITHM, NULL, 0);
    
    BCryptGenerateKeyPair(ActualAlgorithmProvider, &handleToKeyObject, 2048, 0);
    
    BCryptSetProperty(ActualAlgorithmProvider, BCRYPT_PADDING_SCHEMES, (PUCHAR)&temp, sizeof(DWORD), 0);

.
.
.

Unfortunately, BCryptSetProperty return with Invalid handle error.

Answer 1

One problem is incorrect usage of BCryptSetProperty . The BCRYPT_SUPPORTED_PAD_OAEP symbol is not a variable, it's a preprocessor macro.

The documentation for BCRYPT_PADDING_SCHEMES says “data type is a DWORD”, this means the size is 4 bytes.

To set that property, declare a local DWORD variable and pass the address to the function:

DWORD val = BCRYPT_SUPPORTED_PAD_OAEP;
BCryptSetProperty( ActualAlgorithmProvider, BCRYPT_PADDING_SCHEMES, (PUCHAR)(&val), 4, 0 );

Answer 2

Confirm from the developer: BCRYPT_PADDING_SCHEMES is used to retrieve the padding schemes supported by the RSA algorithm provider . If you want to use one of the supported padding schemes(OAEP padding scheme for example), you can specify the BCRYPT_PAD_OAEP flag in BCryptEncrypt / BCryptDecrypt .