Crypto++ AES crashing in release mode using VS2017

Question

I have the following code. A very simple example of encryption and decryption of string "abcd". I've tried it with sample from Crypto++ ( https://www.cryptopp.com/wiki/CBC_mode ) and it produces the same exception.

AutoSeededRandomPool rand;

SecByteBlock key(nullptr, AES::MAX_KEYLENGTH);
rand.GenerateBlock(key, key.size());

byte iv[AES::BLOCKSIZE];
rand.GenerateBlock(iv, AES::BLOCKSIZE);

std::string encryptedData;
CBC_Mode<AES>::Encryption cbcEncryption(key, key.size(), iv);

StringSource ss("abcd", true,
    new StreamTransformationFilter(cbcEncryption,
        new StringSink(encryptedData)
    )
);

std::string decryptedData;

CBC_Mode<AES>::Decryption cbcDecryption(key, key.size(), iv);

StringSource ss2(encryptedData, true,
    new StreamTransformationFilter(cbcDecryption,
        new StringSink(decryptedData)
    )
);

The problem is when I build in debug mode, it works fine but when I do it in release mode I get an exception from Crypto++ code ("StreamTransformationFilter: invalid PKCS #7 block padding found")

Answer 1

The problem is when I build in debug mode, it works fine but when I do it in release mode I get an exception from Crypto++ code ("StreamTransformationFilter: invalid PKCS #7 block padding found") ...

It appears to be a compiler issue related to global optimizations. Our workaround was to disable global optimizations for the source file rijndael.cpp .

In rijndael.cpp you can add the following around the top of the file to avoid the issue:

#if defined(_MSC_VER) && (_MSC_VER >= 1910)
# pragma optimize("", off)
# pragma optimize("ts", on)
#endif

You can reproduce the issue with the following in rijndael.cpp :

#if defined(_MSC_VER) && (_MSC_VER >= 1910)
# pragma optimize("", off)
# pragma optimize("g", on)
#endif

Also see Commit f57df06c5e6d and pragma optimize on MSDN.

---

If your machine has AES-NI but you want to reproduce the issue, then comment-out the code that assigns g_hasAESNI in cpu.cpp . g_hasAESNI will retain the default value of false .

--- a/cpu.cpp
+++ b/cpu.cpp
@@ -242,7 +242,7 @@ void DetectX86Features()
        g_hasSSSE3 = g_hasSSE2 && ((cpuid1[2] & (1<< 9)) != 0);
        g_hasSSE41 = g_hasSSE2 && ((cpuid1[2] & (1<<19)) != 0);
        g_hasSSE42 = g_hasSSE2 && ((cpuid1[2] & (1<<20)) != 0);
-       g_hasAESNI = g_hasSSE2 && ((cpuid1[2] & (1<<25)) != 0);
+       //g_hasAESNI = g_hasSSE2 && ((cpuid1[2] & (1<<25)) != 0);
        g_hasCLMUL = g_hasSSE2 && ((cpuid1[2] & (1<< 1)) != 0);