无法使用Crypto ++执行RSA加密/解密（isValidCoding为false）

Question

I am using Crypto++ to encrypt an array of bytes using RSA. I have followed Crypto++ wiki's samples with no luck getting them to work. Encryption and Decryption in all the samples are done within a single process but I am trying to decrypt the content which is already encrypted in another process. Here is my code:

class FixedRNG : public CryptoPP::RandomNumberGenerator
{
public:
    FixedRNG(CryptoPP::BufferedTransformation &source) : m_source(source) {}

    void GenerateBlock(byte *output, size_t size)
    {
        m_source.Get(output, size);
    }

private:
    CryptoPP::BufferedTransformation &m_source;
};


uint16_t Encrypt()
{
    byte *oaepSeed = new byte[2048];
    for (int i =  0; i < 2048; i++)
    {
        oaepSeed[i] = (byte)i;
    }
    CryptoPP::ByteQueue bq;
    bq.Put(oaepSeed, 2048);
    FixedRNG prng(bq);
    Integer n("Value of N"),
    e("11H"),
    d("Value of D");
    RSA::PrivateKey privKey;
    privKey.Initialize(n, e, d);
    RSA::PublicKey pubKey(privKey);
    CryptoPP::RSAES_OAEP_SHA_Encryptor encryptor( pubKey );
    assert( 0 != encryptor.FixedMaxPlaintextLength() );
    byte blockSize = encryptor.FixedMaxPlaintextLength();
    int divisionCount = fileSize / blockSize;
    int proccessedBytes = 0;
    // Create cipher text space
    uint16_t cipherSize = encryptor.CiphertextLength( blockSize );
    assert( 0 != cipherSize );

    encryptor.Encrypt(prng, (byte*)plaintext, blockSize, (byte*)output);
    return cipherSize;
}

void Decrypt(uint16_t cipherSize)
{
        byte *oaepSeed = new byte[2048];
        for (int i =  0; i < 2048; i++)
        {
            oaepSeed[i] = (byte)i;
        }
        CryptoPP::ByteQueue bq;
        bq.Put(oaepSeed, 2048);
        FixedRNG prng(bq);
        Integer n("Value of N"),
        e("11H"),
        d("Value of D");
        RSA::PrivateKey privKey;
        privKey.Initialize(n, e, d);
        //RSA::PublicKey pubKey(privKey);

        CryptoPP::RSAES_OAEP_SHA_Decryptor decryptor( privKey );

        byte blockSize = decryptor.FixedMaxPlaintextLength();
        assert(blockSize != 0);

        size_t maxPlainTextSize = decryptor.MaxPlaintextLength( cipherSize );
        assert( 0 != maxPlainTextSize );
        void* subBuffer = malloc(maxPlainTextSize);
        CryptoPP::DecodingResult result = decryptor.Decrypt(prng, (byte*)cipherText, cipherSize, (byte*)subBuffer);
        assert( result.isValidCoding );
        assert( result.messageLength <= maxPlainTextSize );
}

Unfortunately, value of isValidCoding is false. I think I am misunderstanding something about RSA encryption/decryption!!
Note that, privKey and pubKey have been validated using KEY.Validate(prng, 3). I have also tried to use RAW RSA instead of OAEP and SHA with no luck. I have tried to debug through crypto++ code, what I am suspicious about is prng variable. I think there is something wrong with it. I have also used AutoSeededRandomPool instead of FixedRNG but it didn't help.
Worth to know that, if I copy the decryption code right after encryption code and execute it in Encrypt() method, everything is fine and isValidCoding is true!!

Answer 1

This is probably not be correct:

byte blockSize = encryptor.FixedMaxPlaintextLength();
...

encryptor.Encrypt(prng, (byte*)plaintext, blockSize, (byte*)output);
return cipherSize;

Try:

size_t maxLength = encryptor.FixedMaxPlaintextLength();
size_t cipherLength = encryptor.CiphertextLength( blockSize );
...

SecureByteBlock secBlock(cipherLength);

cipherLength = encryptor.Encrypt(prng, (byte*)plaintext, blockSize, secBlock);
secBlock.resize(cipherLength);

FixedMaxPlaintextLength returns a size_t , not a byte .

You should probably be calling CiphertextLength on plaintext .

I'm not really sure how you are just returning an uint_t from encrypt() .

You might do better by starting fresh, and using an example from the Crypto++ as a starting point. I'm not sure this design is worth pursuing.

If you start over, then Shoup's Elliptic Curve Integrated Encryption Scheme (ECIES) would be a good choice since it combines public key with symmetric ciphers and authentication tags.