Python-Hex editing specific locations in a file

Question

I want to open up about 135 different offsets in the file in hex form. The sections of interest are the names of the characters skins in the game, so an easy way to edit these and save them would save me MEGA time.

This is code I ended up with, something I could understand. I converted the file to HEX and TEXT form:

import binascii
filename = 'Skin1.pack'
with open(filename, 'rb') as f:
    content = f.read()
out = binascii.hexlify(content)

f = open('hex.txt', 'wb')
f.write(out)
f.close()

import binascii
filename = 'hex.txt'
with open(filename, 'rb') as f:
    content = f.read()

asci = binascii.unhexlify(content) 
w = open('printed-hex.txt', 'wb')
w.write(asci)
w.close()

Now im trying to use this byte to replace some of the text in the file

f = open("printed-hex.txt",'r')
filedata = f.read()
f.close()

newdata = filedata.replace("K n i g h t       ",input)

f = open("printed-hex.txt",'w')
f.write(newdata)
f.close()

but I'm met with this error,

Traceback (most recent call last):
  File "C:\Users\Dee\Desktop\ARC to HEX\Edit-Printed-HEX.py", line 3, in <module>
    filedata = f.read()
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2656: character maps to                                 <undefined>

Answer 1

To nitpick, hex doesn't have 'lines' so you might want to think about how you will limit the location you want to edit. Perhaps edit a fixed number of bytes.

The output you have seen in the console is python attempting to print binary data. It has printed the extended characters because there arn't printable characters that correspond to the characters in the string. You can see that some characters are printable, and that is why you have things like 7(5. in it.

What you need is an easy way to represent the binary data as hex, and a way to convert back. I'll leave the implementation of the actual editor up to you.

import mmap

handle = open('/usr/bin/xxd', 'r')
memorymap = mmap.mmap(handle.fileno(), 0, prot=mmap.PROT_READ)

value_to_hex = dict(enumerate('0123456789ABCDEF'))
hex_to_value = {v: k for (k, v) in value_to_hex.items()}


def expand_byte(byte):
    """ Converts a single byte into 2 4 bit values """
    return [(byte >> s) & 0xF for s in [4, 0]]


def compact_bytes(values):
    """ Converts 2 4 bit values into a single byte """
    return (values[0] << 4) | values[1]


def bin_to_hex(data):
    """ Converts binary data to hex characters """
    return [value_to_hex[v] for b in data for v in expand_byte(b)]


def hex_to_bin(hexadecimal):
    """ Converts hex characters to binary data """
    return [
        compact_bytes([hex_to_value[v] for v in hexadecimal[i:i + 2]])
        for i in range(0, len(hexadecimal), 2)
    ]

test_data = [ord(c) for c in memorymap[0:8]]
hex_data = bin_to_hex(test_data)
final_data = hex_to_bin(hex_data)

print "From '{0}'\nto '{1}'\nto '{2}'".format([chr(c) for c in test_data], hex_data, [chr(c) for c in final_data])

This prints:

From '['\x7f', 'E', 'L', 'F', '\x02', '\x01', '\x01', '\x00']'
to '['7', 'F', '4', '5', '4', 'C', '4', '6', '0', '2', '0', '1', '0', '1', '0', '0']'
to '['\x7f', 'E', 'L', 'F', '\x02', '\x01', '\x01', '\x00']'

Bitwise value manipulation is something you may not have come across before, so you should learn about it. The >> << | and & operators are bitwise operators .

Answer 2

To retrieve the data, operate the mmap object like in the example code;

If you want to open a fragment of data in a hex editor, copy it into a temporary file , then open the file in the editor eg with subprocess.check_call() , then copy the new file's contents back. (That's unless your editor has a command-line option that allows to set focus at a specific offset at startup)

To use just Python's console, use something like

" ".join("%02x"%ord(c) for c in <data>)

to see the data in hex (or just repr to see it in ASCII), or, for more xxd -like look and feel, something 3rd-party like hexview .