Decrypt in chunks a AES 128 CBC encrypted object

Question

I have an Encrypted object in Minio, encrypted using the AES 128 bit CBC algorithm.

The object is quite large (~50 MB) so instead of loading it into the memory completely (which may cause out of memory exception), I am retrieving it in chunks of 1MB. I need to decrypt it before use.

Is it possible to decrypt the object in this way (1MB at a time, the whole object was encrypted in one go)? If yes, how can I do it? I have tried decrypting 16-byte chunks which produce the following errors:

javax.crypto.BadPaddingException: Given final block not properly padded

javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher

Answer 1

Yes, with AES-128-CBC, it is possible to decrypt just a single block of cyphertext. Each block is 128 bits (16 bytes).

See the diagram at https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_block_chaining_(CBC) . As you can see, to decrypt any block of ciphertext, you AES-decrypt the block of ciphertext, then XOR the plaintext with the previous block of ciphertext. (For the first block, the plaintext is XOR'd with the IV).

The library that you are using is probably throwing these exceptions, because it is checking if the decrypted ciphertext is properly padded. Of course, if you are decrypting just one arbitrary block of ciphertext, it will not have the proper padding. However, you can use a tool like openssl to decrypt a single block of ciphertext, given the ciphertext, the key, and the previous block of ciphertext, like so:

echo -n 'bc6d8afc78e805b7ed7551e42da4d877' | xxd -p -r |  openssl aes-128-cbc -d -nopad -K e3e33d2d9591b462c55503f7ec697839 -iv 1d3fa2b7c9008e1cdbc76a1f22388b89

where:

bc6d8afc78e805b7ed7551e42da4d877 is the block of ciphertext that you want to decrypt

e3e33d2d9591b462c55503f7ec697839 is the key

1d3fa2b7c9008e1cdbc76a1f22388b89 is the previous block of ciphertext

Answer 2

To avoid an "out of memory error" you want to decrypt a large (encrypted) file in chunks of 1 mb size - yes it's possible with AES CBC mode.

Below you find a complete example that is generating a sample plaintext file ('plaintext.dat') with random content with the size of 50 mb + 1 byte (the + 1 byte is good to test for file sizes that are not exact multiples of 16 = AES blocksize).

In the next step this file is getting encrypted to 'ciphertext.dat' using a randomly created initialization vector and key.

The last step is the requested decryption method - it decrypts the encrypted file in chunks of 1 mb and in the lines '// obuf holds the decrypted chunk, do what you want to do with the data' and '// final data' you do have the decrypted data in the byte array obuf. For testing I'm writing the decrypted data to the file 'decryptedtext.dat' in appending mode (for that reason this file is deleted in the beginning if it exists).

To prove that decryption was successful I'm comparing the SHA256-hashes of plaintext- and decryptedtext-files.

Two notes: I'm using a 32 byte = 256 bit long key for AES CBC 256. This program has no proper exception handling and is for educational purposes only.

result:

decrypt AES CBC 256 in 1 mb chunks

file with random data created: plaintext.dat
encryption to ciphertext.dat was successfull: true

decryption in chunks of 1 mb
decrypted file written to decryptedtext.dat
plaintext equals decrytedtext file: true

code:

import javax.crypto.*;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.nio.file.Files;
import java.security.*;
import java.util.Arrays;

public class AES_CBC_chunk_decryption {
    public static void main(String[] args) throws IOException, InvalidKeyException, NoSuchAlgorithmException, NoSuchPaddingException,
            InvalidAlgorithmParameterException, BadPaddingException, IllegalBlockSizeException {
        System.out.println("https://stackoverflow.com/questions/63325528/decrypt-in-chunks-a-aes-128-cbc-encrypted-object/63325529#63325529");
        System.out.println("decrypt AES CBC 256 in 1 mb chunks");

        // setup for creation of a 50mb encrypted file
        int filesize = (50 * 1024 * 1024) + 1; // 50 mb + 1 byte = 52428801 bytes
        String filenamePlaintext = "plaintext.dat";
        String filenameCiphertext = "ciphertext.dat";
        String filenameDecryptedtext = "decryptedtext.dat";

        File file = new File("plaintext.dat");
        // fill with random bytes.
        try (FileOutputStream out = new FileOutputStream(file)) {
            byte[] bytes = new byte[filesize];
            new SecureRandom().nextBytes(bytes);
            out.write(bytes);
        }
        System.out.println("\nfile with random data created: " + filenamePlaintext);
        // delete decrypted file if it exists
        Files.deleteIfExists(new File(filenameDecryptedtext).toPath());

        // setup random key & iv
        SecureRandom secureRandom = new SecureRandom();
        byte[] iv = new byte[16];
        byte[] key = new byte[32]; // I'm using a 32 byte = 256 bit long key for aes 256
        secureRandom.nextBytes(iv);
        secureRandom.nextBytes(key);

        // encrypt complete file
        boolean resultEncryption = encryptCbcFileBufferedCipherOutputStream(filenamePlaintext, filenameCiphertext, key, iv);
        System.out.println("encryption to " + filenameCiphertext + " was successfull: " + resultEncryption);
        // encrypted file is 52428816 bytes long

        System.out.println("\ndecryption in chunks of 1 mb");
        // decryption in chunks of 1 mb
        try (FileInputStream in = new FileInputStream(filenameCiphertext)) {
            byte[] ibuf = new byte[(1024 * 1024)]; // chunks of 1 mb
            int len;
            Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
            SecretKeySpec secretKeySpec = new SecretKeySpec(key, "AES");
            IvParameterSpec ivParameterSpec = new IvParameterSpec(iv);
            cipher.init(Cipher.DECRYPT_MODE, secretKeySpec, ivParameterSpec);
            while ((len = in.read(ibuf)) != -1) {
                byte[] obuf = cipher.update(ibuf, 0, len);
                if (obuf != null)
                    // obuf holds the decrypted chunk, do what you want to do with the data
                    // I'm writing it to a file in appending mode
                    try (FileOutputStream output = new FileOutputStream(filenameDecryptedtext, true)) {
                        output.write(obuf);
                    }
            }
            byte[] obuf = cipher.doFinal();
            if (obuf != null)
                // final data
                try (FileOutputStream output = new FileOutputStream(filenameDecryptedtext, true)) {
                    output.write(obuf);
                }
        }
        System.out.println("decrypted file written to " + filenameDecryptedtext);
        System.out.println("plaintext equals decrytedtext file: " + filecompareSha256Large(filenamePlaintext, filenameDecryptedtext));
    }


    public static boolean encryptCbcFileBufferedCipherOutputStream(String inputFilename, String outputFilename, byte[] key, byte[] iv)
            throws IOException, NoSuchPaddingException, NoSuchAlgorithmException, InvalidKeyException, InvalidAlgorithmParameterException {
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        try (FileInputStream in = new FileInputStream(inputFilename);
             FileOutputStream out = new FileOutputStream(outputFilename);
             CipherOutputStream encryptedOutputStream = new CipherOutputStream(out, cipher);) {
            SecretKeySpec secretKeySpec = new SecretKeySpec(key, "AES");
            IvParameterSpec ivParameterSpec = new IvParameterSpec(iv);
            cipher.init(Cipher.ENCRYPT_MODE, secretKeySpec, ivParameterSpec);
            byte[] buffer = new byte[8096];
            int nread;
            while ((nread = in.read(buffer)) > 0) {
                encryptedOutputStream.write(buffer, 0, nread);
            }
            encryptedOutputStream.flush();
        }
        if (new File(outputFilename).exists()) {
            return true;
        } else {
            return false;
        }
    }

    public static boolean filecompareSha256Large(String filename1, String filename2) throws IOException, NoSuchAlgorithmException {
        boolean result = false;
        byte[] hash1 = generateSha256Buffered(filename1);
        byte[] hash2 = generateSha256Buffered(filename2);
        result = Arrays.equals(hash1, hash2);
        return result;
    }

    private static byte[] generateSha256Buffered(String filenameString) throws IOException, NoSuchAlgorithmException {
        // even for large files
        byte[] buffer = new byte[8192];
        int count;
        MessageDigest md = MessageDigest.getInstance("SHA-256");
        BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filenameString));
        while ((count = bis.read(buffer)) > 0) {
            md.update(buffer, 0, count);
        }
        bis.close();
        return md.digest();
    }
}

Answer 3

Yes, it is possible. However, due to the mode and padding it may be trickier to program than it looks at first sight.

However, I've created a class that will happily decode from any offset and to any size. Note that the ciphertext should not contain the IV.

In hindsight I might better have used ByteBuffer to make it a bit more flexible, but yeah, that will require an entire rewrite...

package com.stackexchange.so;

import java.security.GeneralSecurityException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.SecretKey;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;

/**
 * A class that helps you to partially decrypt a CBC ciphertext. Although this class helps you to partially decrypt any
 * part, you'd probably want to decrypt chunks that consists of a specific number of blocks; both the <code>off</code>
 * and <code>len</code> parameter should be a modulus the block size. If you know the exact plaintext length then you
 * can size the last chunk precisely.
 *
 * @author maartenb
 */
public class CBCDecryptByOffset {

    private enum State {
        UNINITIALIZED, INITIALIZED, RUNNING;
    };

    private final Cipher cbcCipher;

    private SecretKey symKey;
    private IvParameterSpec iv;
    
    private State state = State.UNINITIALIZED;

    /**
     * Creates the CBC decryptor class and initializes it.
     * @param blockCipher the block cipher, without block cipher mode or padding indication e.g. <code>"AES"</code>
     * @throws NoSuchAlgorithmException if the block cipher is not available for <code>"CBC"</code>
     * @throws NoSuchPaddingException if the block cipher in CBC mode is not available with <code>"NoPadding"</code> 
     */
    public CBCDecryptByOffset(String blockCipher) throws NoSuchAlgorithmException, NoSuchPaddingException {
        this.cbcCipher = Cipher.getInstance(blockCipher + "/CBC/NoPadding");
    }

    /**
     * Mimics {@link Cipher#init(int, java.security.Key, java.security.spec.AlgorithmParameterSpec)} except that it
     * doesn't include options for encryption, wrapping or unwrapping.
     * 
     * @param symKey the key to use
     * @param iv     the IV to use
     * @throws InvalidKeyException                if the key is not valid for the block cipher
     * @throws InvalidAlgorithmParameterException if the IV is not valid for CBC, i.e. is not the block size
     */
    public void init(SecretKey symKey, IvParameterSpec iv)
            throws InvalidKeyException, InvalidAlgorithmParameterException {
        this.symKey = symKey;
        this.iv = iv;
        // init directly, probably we want to start here, and it will perform a cursory check of the key and IV
        this.cbcCipher.init(Cipher.DECRYPT_MODE, symKey, iv);
        this.state = State.INITIALIZED;
    }

    /**
     * Decrypts a partial number of bytes from a CBC encrypted ciphertext with PKCS#7 compatible padding.
     * 
     * @param fullCT the full ciphertext
     * @param off    the offset within the full ciphertext to start decrypting
     * @param len    the amount of bytes to decrypt
     * @return the plaintext of the partial decryption
     * @throws BadPaddingException       if the ciphertext is not correctly padded (only checked for the final CT block)
     * @throws IllegalBlockSizeException if the ciphertext is empty or not a multiple of the block size
     */
    public byte[] decryptFromOffset(byte[] fullCT, int off, int len)
            throws BadPaddingException, IllegalBlockSizeException {
        if (state == State.UNINITIALIZED) {
            throw new IllegalStateException("Instance should be initialized before decryption");
        }

        int n = cbcCipher.getBlockSize();
        if (fullCT.length == 0 || fullCT.length % n != 0) {
            throw new IllegalBlockSizeException(
                    "Ciphertext must be a multiple of the blocksize, and should contain at least one block");
        }
        if (off < 0 || off > fullCT.length) {
            throw new IllegalArgumentException("Invalid offset: " + off);
        }
        if (len < 0 || off + len < 0 || off + len > fullCT.length) {
            throw new IllegalArgumentException("Invalid len");
        }

        if (len == 0) {
            return new byte[0];
        }

        final int blockToDecryptFirst = off / n;
        final int blockToDecryptLast = (off + len - 1) / n;
        final int bytesToDecrypt = (blockToDecryptLast - blockToDecryptFirst + 1) * n;

        final byte[] pt;
        try {
            // determine the IV to use
            if (state != State.INITIALIZED || off != 0) {
                IvParameterSpec vector;
                final int blockWithVector = blockToDecryptFirst - 1;
                if (blockWithVector == -1) {
                    vector = iv;
                } else {
                    vector = new IvParameterSpec(fullCT, blockWithVector * n, n);
                }

                cbcCipher.init(Cipher.DECRYPT_MODE, symKey, vector);
            }

            // perform the actual decryption (note that offset and length are in bytes)
            pt = cbcCipher.doFinal(fullCT, blockToDecryptFirst * n, bytesToDecrypt);
        } catch (GeneralSecurityException e) {
            throw new RuntimeException("Incorrectly programmed, error should never appear", e);
        }

        // we need to unpad if the last block is the final ciphertext block
        int sigPadValue = 0;
        final int finalCiphertextBlock = (fullCT.length - 1) / n;
        if (blockToDecryptLast == finalCiphertextBlock) {
            int curPaddingByte = bytesToDecrypt - 1;
            int padValue = Byte.toUnsignedInt(pt[curPaddingByte]);
            if (padValue == 0 || padValue > n) {
                throw new BadPaddingException("Invalid padding");
            }
            for (int padOff = curPaddingByte - 1; padOff > curPaddingByte - padValue; padOff--) {
                if (Byte.toUnsignedInt(pt[padOff]) != padValue) {
                    throw new BadPaddingException("Invalid padding");
                }
            }

            // somebody tries to decrypt just padding bytes
            if (off >= (blockToDecryptLast + 1) * n - padValue) {
                sigPadValue = len;
            } else {
                // calculate if any (significant) padding bytes need to be ignored within the plaintext
                int bytesInFinalBlock = (off + len - 1) % n + 1;
                sigPadValue = padValue - (n - bytesInFinalBlock);
                if (sigPadValue < 0) {
                    sigPadValue = 0;
                }
            }
        }

        int ptStart = off - blockToDecryptFirst * n;
        int ptSize = len - sigPadValue;

        state = State.RUNNING;

        if (pt.length == ptSize) {
            return pt;
        }

        return Arrays.copyOfRange(pt, ptStart, ptStart + ptSize);
    }
}

Note that I've tested the general functionality but I'd make sure that I wrap it with some JUnit tests if I were you.