OpenSSL: AES CCM 256 bit encryption of large file by blocks: is it possible?

Question

I am working on a task to encrypt large files with AES CCM mode ( 256-bit key length). Other parameters for encryption are:

• tag size: 8 bytes
• iv size: 12 bytes

Since we already use OpenSSL 1.0.1c I wanted to use it for this task as well.

The size of the files is not known in advance and they can be very large. That's why I wanted to read them by blocks and encrypt each blocks individually with EVP_EncryptUpdate up to the file size.

Unfortunately the encryption works for me only if the whole file is encrypted at once. I get errors from EVP_EncryptUpdate or strange crashes if I attempt to call it multiple times. I tested the encryption on Windows 7 and Ubuntu Linux with gcc 4.7.2.

I was not able to find and information on OpenSSL site that encrypting the data block by block is not possible (or possible).

Additional references:

• http://www.fredriks.se/?p=23
• http://incog-izick.blogspot.in/2011/08/using-openssl-aes-gcm.html

Please see the code below that demonstrates what I attempted to achieve. Unfortunately it is failing where indicated in the for loop.

#include <QByteArray>
#include <openssl/evp.h>

// Key in HEX representation
static const char keyHex[] = "d896d105b05aaec8305d5442166d5232e672f8d5c6dfef6f5bf67f056c4cf420";
static const char ivHex[]  = "71d90ebb12037f90062d4fdb";

// Test patterns
static const char orig1[] = "Very secret message.";

const int c_tagBytes      = 8;
const int c_keyBytes      = 256 / 8;
const int c_ivBytes       = 12;

bool Encrypt()
{
    EVP_CIPHER_CTX *ctx;
    ctx = EVP_CIPHER_CTX_new();
    EVP_CIPHER_CTX_init(ctx);

    QByteArray keyArr = QByteArray::fromHex(keyHex);
    QByteArray ivArr = QByteArray::fromHex(ivHex);

    auto key = reinterpret_cast<const unsigned char*>(keyArr.constData());
    auto iv = reinterpret_cast<const unsigned char*>(ivArr.constData());

    // Initialize the context with the alg only
    bool success = EVP_EncryptInit(ctx, EVP_aes_256_ccm(), nullptr, nullptr);
    if (!success) {
        printf("EVP_EncryptInit failed.\n");
        return success;
    }

    success = EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_CCM_SET_IVLEN, c_ivBytes, nullptr);
    if (!success) {
        printf("EVP_CIPHER_CTX_ctrl(EVP_CTRL_CCM_SET_IVLEN) failed.\n");
        return success;
    }
    success = EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_CCM_SET_TAG, c_tagBytes, nullptr);
    if (!success) {
        printf("EVP_CIPHER_CTX_ctrl(EVP_CTRL_CCM_SET_TAG) failed.\n");
        return success;
    }

    success = EVP_EncryptInit(ctx, nullptr, key, iv);
    if (!success) {
        printf("EVP_EncryptInit failed.\n");
        return success;
    }

    const int bsize = 16;
    const int loops = 5;
    const int finsize = sizeof(orig1)-1; // Don't encrypt '\0'

    // Tell the alg we will encrypt size bytes
    // http://www.fredriks.se/?p=23
    int outl = 0;
    success = EVP_EncryptUpdate(ctx, nullptr, &outl, nullptr, loops*bsize + finsize);
    if (!success) {
        printf("EVP_EncryptUpdate for size failed.\n");
        return success;
    }
    printf("Set input size. outl: %d\n", outl);

    // Additional authentication data (AAD) is not used, but 0 must still be
    // passed to the function call:
    // http://incog-izick.blogspot.in/2011/08/using-openssl-aes-gcm.html
    static const unsigned char aadDummy[] = "dummyaad";
    success = EVP_EncryptUpdate(ctx, nullptr, &outl, aadDummy, 0);
    if (!success) {
        printf("EVP_EncryptUpdate for AAD failed.\n");
        return success;
    }
    printf("Set dummy AAD. outl: %d\n", outl);

    const unsigned char *in = reinterpret_cast<const unsigned char*>(orig1);
    unsigned char out[1000];
    int len;

    // Simulate multiple input data blocks (for example reading from file)
    for (int i = 0; i < loops; ++i) {
        // ** This function fails ***
        if (!EVP_EncryptUpdate(ctx, out+outl, &len, in, bsize)) {
            printf("DHAesDevice: EVP_EncryptUpdate failed.\n");
            return false;
        }
        outl += len;
    }

    if (!EVP_EncryptUpdate(ctx, out+outl, &len, in, finsize)) {
        printf("DHAesDevice: EVP_EncryptUpdate failed.\n");
        return false;
    }
    outl += len;

    int finlen;
    // Finish with encryption
    if (!EVP_EncryptFinal(ctx, out + outl, &finlen)) {
        printf("DHAesDevice: EVP_EncryptFinal failed.\n");
        return false;
    }
    outl += finlen;
    // Append the tag to the end of the encrypted output
    if (!EVP_CIPHER_CTX_ctrl(ctx, EVP_CTRL_CCM_GET_TAG, c_tagBytes, out + outl)) {
        printf("DHAesDevice: EVP_CIPHER_CTX_ctrl failed.\n");
        return false;
    };
    outl += c_tagBytes;
    out[outl] = '\0';

    EVP_CIPHER_CTX_cleanup(ctx);
    EVP_CIPHER_CTX_free(ctx);

    QByteArray enc(reinterpret_cast<const char*>(out));

    printf("Plain text size: %d\n", loops*bsize + finsize);
    printf("Encrypted data size: %d\n", outl);

    printf("Encrypted data: %s\n", enc.toBase64().data());

    return true;
}

EDIT (Wrong Solution)

The feedback that I received made me think in a different direction and I discovered that EVP_EncryptUpdate for size must be called for each block that it being encrypted, not for the total size of the file . I moved it just before the block is encrypted: like this:

for (int i = 0; i < loops; ++i) {
    int buflen;
    (void)EVP_EncryptUpdate(m_ctx, nullptr, &buflen, nullptr, bsize);
    // Resize the output buffer to buflen here
    // ...
    // Encrypt into target buffer
    (void)EVP_EncryptUpdate(m_ctx, out, &len, in, buflen);
    outl += len;
}

AES CCM encryption block by block works this way, but not correctly , because each block is treated as independent message.

EDIT 2

OpenSSL's implementation works properly only if the complete message is encrypted at once.

http://marc.info/?t=136256200100001&r=1&w=1

I decided to use Crypto++ instead.

Answer 1

For AEAD-CCM mode you cannot encrypt data after associated data was feed to the context. Encrypt all the data, and only after it pass the associated data.

Answer 2

I found some mis-conceptions here

first of all EVP_EncryptUpdate(ctx, nullptr, &outl calling this way is to know how much output buffer is needed so you can allocate buffer and second time give the second argument as valid big enough buffer to hold the data.

You are also passing wrong (over written by previous call) values when you actually add the encrypted output.