RSA-OAEP with SHA -256 key size 2048 bits using OpenSSL

Question

I am trying to address a use case exactly same as How to encrypt data using RSA, with SHA-256 as hash function and MGF1 as mask generating function? , but I need a few more clarity on this.

The above query was raised in the year 2013. At that time the OpenSSL only supported SHA1 hash (hard coded) for OAEP padding. In the latest OpenSSL (1.0.2k), I can see that this is addressed by using the following API:

int RSA_padding_add_PKCS1_OAEP_mgf1(unsigned char *to, int tlen,
                                    const unsigned char *from, int flen,
                                    const unsigned char *param, int plen,
                                    const EVP_MD *md, const EVP_MD mgf1md)

RSA_public_encrypt() does not take EVP_MD structure as argument I'm not sure how to specify it.

How can I invoke the SHA-256 mode in RSA_public_encrypt() with a mask generation function?

Answer 1

RSA_public_encrypt(...) is deprecated; EVP_PKEY_encrypt(EVP_PKEY_CTX *ctx, ...) should be used instead.

Padding, mask generation function and other parameters are configured for the context which is passed as the first argument to EVP_PKEY_encrypt:

    EVP_PKEY* evp_key = PEM_read_bio_PUBKEY(bio, NULL, NULL, NULL);
    if (evp_key == NULL) {
        // handle error
    }

    EVP_PKEY_CTX* ctx = EVP_PKEY_CTX_new(evp_key, NULL);
    if (ctx == NULL) {
        // handle error
    }

    if (EVP_PKEY_encrypt_init(ctx) <= 0) {
        // handle error
    }

    if (EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_OAEP_PADDING) <= 0) {
        // handle error
    }
    if (EVP_PKEY_CTX_set_rsa_oaep_md(ctx, EVP_sha256()) <= 0) {
        // handle error
    }
    if (EVP_PKEY_CTX_set_rsa_mgf1_md(ctx, EVP_sha256()) <= 0) {
        // handle error
    }

    if (EVP_PKEY_encrypt(ctx, encrypted, &outlen, data, len) <= 0) {
        // handle error
    }