Bouncy Castle is throwing "mac check in OCB failed" when trying to decrypt an AES message

Question

I need to decrypt an AES message. I was able to make it work on python(using the pycryptodome library) but I wasn't successful on kotlin/java. Since the decryption mode is OCB and Java does not support it by default, I had to use the Bouncy Castle library.

Here's the working python code:

def decrypt_aes_message(shared_key, encrypted_message):
    encrypted_msg = b64decode(encrypted_message["encryptedMessage"].encode())
    tag = b64decode(encrypted_message["tag"].encode())
    nonce = b64decode(encrypted_message["nonce"].encode())
    cipher = AES.new(shared_key.encode(), AES.MODE_OCB, nonce=nonce)
    return cipher.decrypt_and_verify(encrypted_msg, tag).decode()

Here's the java code:

fun decryptAesMessage2(sharedKey: String, encryptedMessageData: Map<String, String>): ByteArray {
    var encryptedMessage = encryptedMessageData["encryptedMessage"]!!.utf8Base64Decode()
    var tag = encryptedMessageData["tag"]!!.utf8Base64Decode()
    var nonce = encryptedMessageData["nonce"]!!.utf8Base64Decode()

    var key = KeyParameter(sharedKey.toByteArray(Charsets.UTF_8))
    var params = AEADParameters(key, tag.size*8, nonce)
    var cipher = OCBBlockCipher(AESEngine(), AESEngine())

    cipher.init(false, params)

    val out = ByteArray(cipher.getOutputSize(encryptedMessage.size))
    var offset = cipher.processBytes(encryptedMessage, 0, encryptedMessage.size, out, 0)
    offset += cipher.doFinal(out, offset) // Throwing exception here

    return out
}

The java code is throwing the exception org.bouncycastle.crypto.InvalidCipherTextException: mac check in OCB failed on cipher.doFinal

The file debug.zip has the complete problem reproducer. Inside the zip file you'll find:

• py_working_code.py - the working python script(needs pycryptodome in order to work. You can install pycryptodome using pip install pycryptodome )
• bc-debug - gradle project reproducing the problem

Answer 1

There are two issues, one is a bug in your Kotlin code, the other is a bug in one of the two libraries:

Bug in Kotlin code

While PyCryptodome processes ciphertext and tag separately, BC/Kotlin expects the concatenation of both in the order: ciphertext|tag .
Therefore the line encryptedMessage += tag must be added in the Kotlin code:

fun decryptAesMessage2(sharedKey: String, encryptedMessageData: Map<String, String>): ByteArray {
    var encryptedMessage = encryptedMessageData["encryptedMessage"]!!.utf8Base64Decode()
    var tag = encryptedMessageData["tag"]!!.utf8Base64Decode()
    encryptedMessage += tag // Fix
    var nonce = encryptedMessageData["nonce"]!!.utf8Base64Decode()

    var key = KeyParameter(sharedKey.toByteArray(Charsets.UTF_8))
    var params = AEADParameters(key, tag.size*8, nonce)
    var cipher = OCBBlockCipher(AESEngine(), AESEngine())

    cipher.init(false, params)

    val out = ByteArray(cipher.getOutputSize(encryptedMessage.size))
    var offset = cipher.processBytes(encryptedMessage, 0, encryptedMessage.size, out, 0)
    offset += cipher.doFinal(out, offset) // Throwing exception here

    return out
}

---

Test: Below, identical test data is successfully decrypted using the Python code and the fixed Kotlin code:

Python:

encrypted_message = {
  'encryptedMessage': 'LzoelJ9Nv4cruj0JUlxFrNR+mqyO2rvwqDHYwnj0OkvJ+BBvug+ORYVkxA==', 
  'tag': 'hl56drXePWiLkVavVwF3/w==', 
  'nonce': b64encode(b'012345678901').decode()
}
dt = decrypt_aes_message('01234567890123456789012345678901', encrypted_message)
print(dt) # The quick brown fox jumps over the lazy dog

Kotlin:

val encrypted_message = mutableMapOf<String, String>()
encrypted_message["encryptedMessage"] = "LzoelJ9Nv4cruj0JUlxFrNR+mqyO2rvwqDHYwnj0OkvJ+BBvug+ORYVkxA=="
encrypted_message["tag"] = "hl56drXePWiLkVavVwF3/w=="
encrypted_message["nonce"] = Base64.getEncoder().encodeToString("012345678901".toByteArray(Charsets.UTF_8))
val dt = decryptAesMessage2("01234567890123456789012345678901", encrypted_message)
println(String(dt, Charsets.UTF_8)) // The quick brown fox jumps over the lazy dog

Bug in one of the two libraries

Another problem is that both implementations produce different results for a nonce length of 120 bits (15 bytes), the maximum allowed nonce length for OCB (see RFC 7253, 4.2. Encryption: OCB-ENCRYPT ):

The following test uses a fixed key and plaintext and compares the results between the Python and BC/Kotlin code for nonce lengths 13, 14, and 15 bytes:

Key: b'01234567890123456789012345678901'
Plaintext: b'testmessage'

Nonce Länge     13 bytes                                                    14 bytes                                                    15 bytes
Nonce           b'0123456789012'                                            b'01234567890123'                                           b'012345678901234'
Python (ct|tag) 0xb35a69a245ab18fe3b6bae38b179c2a43b341f67c0451256b76bd7    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e
BC/Kotlin       0xb35a69a245ab18fe3b6bae38b179c2a43b341f67c0451256b76bd7    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e    0xa4355068324065f2ad194b058bdb86caa67c225b99021dbd588034

The Python implementation returns the same ciphertext/tag for nonce lengths 14 and 15, while the results for BC/Kotlin differ. This indicates a bug in the Python implementation.

Unfortunately, RFC 7253, Appendix A. Sample Results only provides test vectors whose nonces are all 12 bytes in size, so the bug cannot be assigned more clearly.

Ie if you use a 15 bytes nonce, both implementations are not compatible, where the problem is most likely caused by the Python implementation.