Decrypting using RSACryptoServiceProvider.Decrypt

Question

I'm trying to decrypt a message as part of a key exchange. I've got a 2048 bit RSA private key which I used to generate a certificate. I receive a message as part of a HTTP request which I need to decrypt with my private key. However, I receive the following error message when executing the last line:

"The data to be decrypted exceeds the maximum for this modulus of 256 bytes."

I've tried reducing the byte array of the data to decrypt as well as reversing it. If I do any of those two, I receive a "Bad Data" error.

Any help would be greatly appreciated.

Example of message to decode:

ajJDR09EQkUzT0prRHJlM2I1bzZGYjlaUWFpQTB6U2pQb0JGeDBvQ0tseEpYMGhmUkdSU0VJRnFnOEdQTDV5SlRJZmxoQUYzeFAxS3NGM1hFSnBobGl3Z3Y2UStydkY3ZkgvVmRLSit6bE5MZ3RTN0twUWZUaUZqMjlkLzBGVWVhL25qdnFXYTVrdlBrYUN2T2grZ1Rnc3FEd3U4ZVZiOUxhWVUzQWpRODk3MFY4VjM5c1VWYXRLcXdZbitQQkV4cFFSYXRJUlcyS2taSXpuRGZTVCt3dGZRcHMwU1lra3ZENSt6VHZnSGFRSmZNQXMvUlRiSERPVTZrNWo5dVR3SXNTOCtlalBWYjdMc1phOXU1c1plVTZpTlhvOUp1emxDalZpaVk3YnY0SkJCcHhqclRPaVA4NVhUYWg1TVhRYUZsMTZOVzE4dDMzYndnQmVkQmRwNEN3PT0=

C# code:

        //http request containing the HMAC key which is encrypted against the public key
        hmacKey = oCtx.RequestContext.RequestMessage.ToString();
        hmacKey = hmacKey.Remove(0, 8);
        hmacKey = hmacKey.Remove(hmacKey.Length - 9);

        //decode into binary using Base64
        byte[] data = Convert.FromBase64String(hmacKey);

        string publicCert = "-----BEGIN CERTIFICATE-----......-----END CERTIFICATE-----";
        string privateKey = "-----BEGIN RSA PRIVATE KEY-----......-----END RSA PRIVATE KEY-----";

        byte[] certBuffer = Helpers.GetBytesFromPEM(publicCert, PemStringType.Certificate);
        byte[] keyBuffer = Helpers.GetBytesFromPEM(privateKey, PemStringType.RsaPrivateKey);

        X509Certificate2 x509cert = new X509Certificate2(certBuffer);

        RSACryptoServiceProvider prov = Crypto.DecodeRsaPrivateKey(keyBuffer);
        x509cert.PrivateKey = prov;

        //tried to reduce the size of the data to decrypt as well as reversing it
        //Array.Resize(ref data, 32);
        //Array.Reverse(data);

        byte[] result = prov.Decrypt(data, false);

More info on the GetBytesFromPEM method is available from this example: http://www.codeproject.com/Articles/162194/Certificates-to-DB-and-Back

UPDATE:

Trying to decode twice, I get the following result:

code:

        .....
        byte[] data2 = Convert.FromBase64String(hmacKey);
        string abc = Encoding.Default.GetString(data2);
        byte[] data = Convert.FromBase64String(abc);
        .....
        byte[] result = prov.Decrypt(data, false);
        string result2 = Encoding.Default.GetString(result);

result:

Óh@-šÚz;CÏ7 .«™"ã®ÿRè±àyéK.

Answer 1

The errors are basically due to encoding errors, both binary encoding (base 64) issues and character encoding issues (UTF-8/UTF-16).

1. Usually you would expect a binary HMAC to be encrypted. Instead the HMAC was hex encoded, which in turn was encoded using ASCII encoding (which is compatible with UTF-8). The .NET default is however UTF-16LE (what .NET incorrectly calls Unicode encoding).

2. The resulting ciphertext was base 64 encoded, which is what you would expect if the result needs to be transported in text. Instead double base 64 seemed to have been utilized. As the base 64 decoding resulted in another base 64 encoded string, the result was too large for the RSA decryption to handle.