SQL Server : CLR RijndaelManaged bytes not decrypting correctly from varbinary(max)

Question

I'm trying to encrypt/decrypt documents into a table using a stored procedure, so I created a CLR assembly with encrypt/decrypt functions that use the RijndaelManaged class. I'm able to encrypt the bytes, but when I decrypt the bytes and save the document, I'm noticing there's a difference in encoding which breaks the document. I'm sending the varbinary(max) bytes directly to the encrypt/decrypt function so I'm not sure what's causing a different encoding. I'm wondering how I can get this to decrypt in the correct encoding?

Here's what my assembly looks like:

    public static byte[] AES_EncryptBytes(byte[] input, string pass)
    {
        try
        {
            return EncryptBytesToBytes(input, System.Text.Encoding.UTF8.GetBytes(pass));
        }
        catch (Exception)
        {
            return null;
        }
    }

    public static byte[] AES_DecryptBytes(byte[] input, string pass)
    {
        try
        {
            return DecryptBytesFromBytes(input, System.Text.Encoding.UTF8.GetBytes(pass));
        }
        catch (Exception)
        {
            return null;
        }
    }

    private static byte[] EncryptBytesToBytes(byte[] Input, byte[] Key)
    {
        return EncryptBytesToBytes(Input, Key, null);
    }

    private static byte[] EncryptBytesToBytes(byte[] Input, byte[] Key, byte[] IV)
    {
        // Check arguments.
        if ((Input == null) || (Input.Length <= 0))
        {
            throw (new ArgumentNullException("plainText"));
        }

        if ((Key == null) || (Key.Length <= 0))
        {
            throw (new ArgumentNullException("Key"));
        }

        // Create an RijndaelManaged object
        // with the specified key and IV.
        RijndaelManaged rijAlg = new RijndaelManaged();
        rijAlg.Key = Key;

        if (!(IV == null))
        {
            if (IV.Length > 0)
            {
                rijAlg.IV = IV;
            }
            else
            {
                rijAlg.Mode = CipherMode.ECB;
            }
        }
        else
        {
            rijAlg.Mode = CipherMode.ECB;
        }

        byte[] encrypted = null;
        // Create a decrytor to perform the stream transform.
        ICryptoTransform encryptor = rijAlg.CreateEncryptor(rijAlg.Key, rijAlg.IV);

        encrypted = encryptor.TransformFinalBlock(Input, 0, Input.Length);
        // Return the encrypted bytes from the memory stream.
        return encrypted;
    }

    private static byte[] DecryptBytesFromBytes(byte[] cipherText, byte[] Key)
    {
        return DecryptBytesFromBytes(cipherText, Key, null);
    }

    private static byte[] DecryptBytesFromBytes(byte[] cipherText, byte[] Key, byte[] IV)
    {
        // Check arguments.
        if ((cipherText == null) || (cipherText.Length <= 0))
        {
            throw (new ArgumentNullException("cipherText"));
        }

        if ((Key == null) || (Key.Length <= 0))
        {
            throw (new ArgumentNullException("Key"));
        }

        // Create an RijndaelManaged object
        // with the specified key and IV.
        RijndaelManaged rijAlg = new RijndaelManaged();
        rijAlg.Key = Key;

        if (!(IV == null))
        {
            if (IV.Length > 0)
            {
                rijAlg.IV = IV;
            }
            else
            {
                rijAlg.Mode = CipherMode.ECB;
            }
        }
        else
        {
            rijAlg.Mode = CipherMode.ECB;
        }

        byte[] output = null;
        // Create a decrytor to perform the stream transform.
        ICryptoTransform decryptor = rijAlg.CreateDecryptor(rijAlg.Key, rijAlg.IV);
        // Create the streams used for decryption.
        MemoryStream msDecrypt = new MemoryStream(cipherText);
        CryptoStream csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read);

        StreamReader srDecrypt = new StreamReader(csDecrypt);
        // Read the decrypted bytes from the decrypting stream
        // and place them in a string.
        MemoryStream ms = new MemoryStream();

        while (!srDecrypt.EndOfStream)
        {
            ms.WriteByte((byte)(srDecrypt.Read()));
        }

        ms.Position = 0;
        output = ms.ToArray();
        return output;
    }

Here's what my functions look like:

CREATE FUNCTION [dbo].EncryptBytes
     (@Input VARBINARY(MAX), @KEY [NVARCHAR](100))
RETURNS VARBINARY(MAX) 
WITH EXECUTE AS CALLER
AS 
EXTERNAL NAME [DocumentsEncryption].[AES_EncryptDecrypt.AES_EncryptDecryptLibrary].AES_EncryptBytes
GO

CREATE FUNCTION [dbo].[DecryptBytes]
    (@Input VARBINARY(MAX), @KEY [NVARCHAR](100))
RETURNS VARBINARY(MAX) 
WITH EXECUTE AS CALLER
AS
EXTERNAL NAME [DocumentsEncryption].[AES_EncryptDecrypt.AES_EncryptDecryptLibrary].[AES_DecryptBytes]
GO

And for example, how this is executed:

DECLARE @DocumentStream VARBINARY(MAX)
--these bytes below represent a document
SET @DocumentStream = 0x255044462D312E350D25E2E3CFD30D0A

DECLARE @EncryptionKey NVARCHAR(100)
SET @EncryptionKey = 'ayb&e#i&BWLGMe2V'

DECLARE @EncryptedDocumentStream VARBINARY(MAX)
SET @EncryptedDocumentStream  = dbo.[EncryptBytes](@DocumentStream, @EncryptionKey)

DECLARE @DecryptedDocumentStream VARBINARY(MAX)
SET @DecryptedDocumentStream = dbo.[DecryptBytes](@EncryptedDocumentStream,@EncryptionKey)

--@DecryptedDocumentStream will return the decrypted bytes but the encoding is wrong
SELECT @DecryptedDocumentStream
--This will return:               0x255044462D312E350D25FDFDFDFD0D0A
--Instead of the original bytes:  0x255044462D312E350D25E2E3CFD30D0A

Answer 1

The problem is "somewhere" within all of that stream handling code in your decrypt method. I say that because I'm not going to dig in and find the exact fault. The first thing that leapt out is that your encrypt and decrypt methods don't look "symmetrical" - doing approximately the same things as each other (but some operations reversed). That's usually a bad sign with pairs of encryption/decryption methods ¹ .

So if I make decrypt look like encrypt and don't do all of the mucking about with streams:

private static byte[] DecryptBytesFromBytes(byte[] cipherText, byte[] Key, byte[] IV)
{
  if ((cipherText == null) || (cipherText.Length <= 0))
  {
    throw (new ArgumentNullException("cipherText"));
  }

  if ((Key == null) || (Key.Length <= 0))
  {
    throw (new ArgumentNullException("Key"));
  }

  RijndaelManaged rijAlg = new RijndaelManaged();
  rijAlg.Key = Key;

  if (!(IV == null))
  {
    if (IV.Length > 0)
    {
      rijAlg.IV = IV;
    }
    else
    {
      rijAlg.Mode = CipherMode.ECB;
    }
  }
  else
  {
    rijAlg.Mode = CipherMode.ECB;
  }

  ICryptoTransform decryptor = rijAlg.CreateDecryptor(rijAlg.Key, rijAlg.IV);
  return decryptor.TransformFinalBlock(cipherText, 0, cipherText.Length);
}

(I skipped having an output variable too - I didn't see the need for it, nor comments just telling us what the code is doing, which we can determine by reading the code).

Now this (paired with EncryptBytesToBytes in your question) can successfully round-trip the sample data:

static void Main()
{
  var inp = new byte[] { 0x25, 0x50, 0x44, 0x46, 0x2D, 0x31, 0x2E, 0x35,
                         0x0D, 0x25, 0xE2, 0xE3, 0xCF, 0xD3, 0x0D, 0x0A };
  var key = "ayb&e#i&BWLGMe2V";

  var oup = AES_DecryptBytes(AES_EncryptBytes(inp, key), key);
  Console.ReadLine();
}

By eye, inp and oup contain the same data.

(Insert usual caveats about ECB being a terrible mode to use unless it's been chosen for very specific good reasons)

---

¹ My usual recommendation if you're going to build up a pair of encryption/decryption methods is to do is slowly and simply and make sure that the pair can round-trip at each stage before you add more complexity.

The first stage would just be "returns the input buffer, ignores the key and IV". Write some unit tests that confirms it round trips with some decent size buffers and a specific key and IV.

Then add just a little more complexity to the implementation and check that the unit tests still pass, and iterate until the methods do what you want/need them to do.

If you need "encrypt in one language, decrypt in the other", I'd actually recommend doing all of this twice, in both languages so that they both have both sets of methods. Then verify that the outputs at each stage match between your implementations.