What is the equivalent of my openssl cmdline in C?

Question

I'm trying the implement the following openssl command line in C:

openssl enc -aes-256-cbc -d -in /tmp/out_enc -K \
    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA \
    -iv 0 -nopad -p

Output of the command line:

key=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
iv =00000000000000000000000000000000
<CONNECTION_REQUEST ATS_ID="2" ATP_SEQ_ID="1" REGISTRATION_ID="Y2G5R52S8PP6YX47" SERIAL_NUMBER="724574802" SPC_PRODUCT_TITLE="SPC5300" SPC_FW_VERSION="3.8.5 - R.31629" ATS_NAME="Syst&#232;me (ATS) 2" ATP1_ID="2" ATP1_UID="34" ATP1_NAME="Principal ATP 1" ATP1_COMMS_INTERFACE="1" ATP1_DEST="1, 192.168.1.62:52000" ATP1_CATEGORY="50"/>�z

This is my equivalent in C:

long
_ast_crypt_decrypt_generic(unsigned char* ciphertext, long cipherlen, unsigned char* plaintext, const EVP_CIPHER *cipher) {
    long result = 0;

     /* A 256 bit key */
unsigned char *key = (unsigned char *)"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";

/* A 128 bit IV */
unsigned char *iv = (unsigned char *)"0000000000000000";
    EVP_CIPHER_CTX *ctx = NULL;
    int len;
    long plaintext_len;

    if (cipher) {
        /* Create and initialise the context */
        if (!(ctx = EVP_CIPHER_CTX_new())) {
            g_warning("AstCrypt : EVP_CIPHER_CTX_new failed");
            goto end;
        }

        if (1 != EVP_DecryptInit_ex(ctx, cipher, NULL, key, iv)) {
            g_warning("AstCrypt : EVP_DecryptInit_ex");
        }

        EVP_CIPHER_CTX_set_padding(ctx, 0);

        if (1 != EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, cipherlen)) {
            g_warning("AstCrypt : EVP_DecryptUpdate");
            goto end;
        }
        plaintext_len = len;

        if (1 != EVP_DecryptFinal_ex(ctx, plaintext + len, &len)) {
             g_warning("AstCrypt : EVP_DecryptFinal_ex failed");
             goto end;
        }
        plaintext_len += len;

        result= plaintext_len;
    } 
    else {
        g_warning("AstCrypt : Failed to get the openssl elements.");
    }

end:
    if (ctx)
        EVP_CIPHER_CTX_free(ctx);

    return result;
}

long plainSize = _ast_crypt_decrypt_generic(headerData, headerLength, bPlain, EVP_aes_256_cbc());
 //fwrite(bPlain, plainSize, 1, stdout);

The previous code sample doesn't decrypt the data correctly (in C).

Do you have any ideas what I am missing?

It's probably a missing understanding of the key/iv format in the openssl lib.

Answer 1

The answer is that the key/iv must be represented in hex :

The following works as expected :

unsigned char key[] =
        {   0xAA, 0xAA, 0xAA, 0xAA,
        0xAA, 0xAA, 0xAA, 0xAA,
        0xAA, 0xAA, 0xAA, 0xAA,
        0xAA, 0xAA, 0xAA, 0xAA,
        0xAA, 0xAA, 0xAA, 0xAA,
0xAA, 0xAA, 0xAA, 0xAA,
0xAA, 0xAA, 0xAA, 0xAA,
0xAA, 0xAA, 0xAA, 0xAA,0 };
//unsigned char key[] = {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0} ;
    /* A 128 bit IV */
    unsigned char iv[] =
        {   0x00, 0x00, 0x00, 0x00,
        0x00, 0x00, 0x00, 0x00,
        0x00, 0x00, 0x00, 0x00,
        0x00, 0x00, 0x00, 0x00,0 };

Answer 2

The problem is that you are assuming the key is direct copy of the "password". What you are doing with that command line is specifying the key as a hex string. So what you need to do is convert your "password" from a hex string into a the key buffer. In your example you should end up with every byte set to the hex value "0xAA".

Because you are passing in the key directly, there is no need for the MD5 parameter (to your function or into openssl).

If you wanted to using a password instead of a "hex string", then you need to hash it somehow into the key buffer. This is where the MD5 parameter comes in. You can use something like PBKDF2 (although I would use the openssl default SHA256 and not MD5 as the hashing function).
eg openssl enc -aes-256-cbc -pbkdf2 -k password