Correctly add string to a REG_BINARY type in Windows Registry

Question

I am trying to automate the process of adding software policy hash rules to Windows, and am currently having a problem adding valid hashes to the registry. This code creates a key and adds the hash to the registry:

HKEY* m_hKey;
string md5Digest;
string valueName = "ItemData";
vector<BYTE> itemData;

/*
use Crypto++ to get file hash
*/

//convert string to format that can be loaded into registry
for (int i = 1; i < md5Digest.length(); i += 2)
    itemData.push_back('0x' + md5Digest[i - 1] + md5Digest[i]);

// Total data size, in bytes
const DWORD dataSize = static_cast<DWORD>(itemData.size());

::RegSetValueEx(
    m_hKey,
    valueName.c_str(),
    0, // reserved
    REG_BINARY,
    &itemData[0],
    dataSize
);

This works fine, and adds the key to the registry:

But when comparing the registry key to a rule added by Group Policy you can see a very important difference:

The 'ItemData' values are different between them. The bottom picture's ItemData value is the correct output. When debugging the program I can clearly see that md5Digest has the correct value, so the problem lies with the conversion of the md5Digest string to the ItemData vector of BYTEs or unsigned chars....

What is the problem with my code, why is the data being entered incorrectly to the registry?

Answer 1

You have '0x' two-letter char literal summed up with md5Digest[i - 1] + md5Digest[i] and then trunketed to BYTE . This looked like you were trying to build "0xFF" byte value out of them. You should store md5 string directly:

const DWORD dataSize = static_cast<DWORD>(md5Digest.size());

::RegSetValueEx(
    m_hKey,
    valueName.c_str(),
    0, // reserved
    REG_BINARY,
    reinterpret_cast< BYTE const *>(md5Digest.data()),
    dataSize
);

If you actually need to store binary representation of hex numbers from md5 then you need to convert them to bytes like this:

BYTE char_to_halfbyte(char const c)
{
    if(('0' <= c) && (c <= '9'))
    {
        return(static_cast<BYTE>(c - `0`));
    }
    else
    {
        assert(('A' <= c) && (c <= 'F'));
        return(static_cast<BYTE>(10 + c - `A`));
    }
}

for(std::size_t i = 0; i < md5Digest.length(); i += 2)
{
    assert((i + 1) < md5Digest.length());
    itemData.push_back
    (
       (char_to_halfbyte(md5Digest[i    ]) << 4)
       |
       (char_to_halfbyte(md5Digest[i + 1])     )
    );
}

Answer 2

You have a string that you want to convert to a byte array. You can write a helper function to convert 2 chars to a BYTE:

using BYTE = unsigned char;

BYTE convert(char a, char b)
{
    // Convert hex char to byte
    // Needs fixin for lower case
    if (a >= '0' && a <= '9') a -= '0';
    else a -= 55;  // 55 = 'A' - 10
    if (b >= '0' && b <= '9') b -= '0';
    else b -= 55;

    return (a << 4) | b;
}
....
vector<BYTE> v;
string s = "3D65B8EBDD0E";
for (int i = 0; i < s.length(); i+=2) {
    v.push_back(convert(s[i], s[i+1]));
}

v now contains {0x3D, 0x65, 0xB8, 0xEB, 0xDD, 0x0E}

Or, as mention by @RbMm, you can use the CryptStringToBinary Windows function:

#include <wincrypt.h>
...
std::string s = "3D65B8EBDD0E";
DWORD hex_len = s.length() / 2;
BYTE *buffer = new BYTE[hex_len];
CryptStringToBinary(s.c_str(),
    s.length(),
    CRYPT_STRING_HEX,
    buffer, 
    &hex_len,
    NULL,
    NULL
    );