如何像Python 2.7中那样从字节字符串中提取混合的二进制和ascii值？

Question

The following represents a binary image extracted from a file (spaces inserted between bytes to make reading easier). File is opened with 'rb' mode.

01 77 33 9F 41 42 43 44 00 11 11 11

In Python 2.7, I read it as a character string and I use ord() to extract the binary values and then I can extract or even search the string for a specific text value (such as the "ABCD" in characters 4-7). The binary bytes can be anything from 0-FF. I've been putting off conversion to python 3 partly because of this.

I need to be able, in Python 3, to treat a string of bytes as a mixture of binary and ascii (not unicode) values. The format is not fixed, it consists of data structures. For example, the 33 in byte 2 might be a record length that tells me where the start of the next record is. In other words, I can't just say that I know the text string is always in location 4.

I don't write the file, I just use it, so changing it is not an option.

I've seen lots of examples of using b' and other things to convert fixed strings but I need a way to intermix these values, extracting bytes, 2-byte to 8-byte values as 16-bit to 64-bit words, and extracting/searching for ASCII strings within the larger string.

The byte/character separation in Python 3 seems somewhat inflexible for what I need. I'm sure there's a way to do this I just haven't found an example or an answered question that seems to cover this case.

This is a simplified example, I can't provide real data (it's proprietary) but this illustrates the problem. The real files may be short (<1K) or huge (>100K), containing multiple records of different sizes.

Is there an easy, straightforward way to essentially replicate the functionality I have in Python 2.7?

This is on Windows.

Thanks

Answer 1

I need to be able, in Python 3, to treat a string of bytes as a mixture of binary and ascii (not unicode) values. The format is not fixed, it consists of data structures. For example, the 33 in byte 2 might be a record length that tells me where the start of the next record is. In other words, I can't just say that I know the text string is always in location 4.

1. Read the file in binary mode, as you are doing. This produces a bytes object, which in 3.x is not the same as a str (as it would be in 2.x).

2. Interpret the bytes as bytes, as needed, to figure out the general structure of the data. Slicing the bytes produces another bytes as before; indexing produces an int with the numeric value of that single byte ( not as before) - no ord required.

3. When you have determined a subset of the bytes that represent a string (let's say for convenience that you have sliced it out), convert to string using the appropriate encoding : eg str(my_bytes, 'ascii') . Note that ASCII will not handle byte values 0x80 through 0xFF; especially with binary-ish legacy file formats, there's a good chance your data is actually something like Latin-1: str(my_bytes, 'iso-8859-1') .

search the string for a specific text value

You can search at either the text or the byte level - bytes objects support the in operator, searching for either a subsequence of bytes or a single integer value. Whether it makes more sense to search before or after string conversion will depend on what you are doing.

using b' and other things to convert fixed strings

b'' is just the syntax for a literal bytes object. It's what you'll see if you ask for the repr of what you read from the file. Prefixing a b onto an existing string literal in your code isn't really "converting" anything, but replacing it with the value you should have had in the first place.

2-byte to 8-byte values as 16-bit to 64-bit words

The documentation says it at least as well as I could:

>>> help(int.from_bytes)
Help on built-in function from_bytes:

from_bytes(...) method of builtins.type instance
    int.from_bytes(bytes, byteorder, *, signed=False) -> int

    Return the integer represented by the given array of bytes.

    The bytes argument must be a bytes-like object (e.g. bytes or bytearray).

    The byteorder argument determines the byte order used to represent the
    integer.  If byteorder is 'big', the most significant byte is at the
    beginning of the byte array.  If byteorder is 'little', the most
    significant byte is at the end of the byte array.  To request the native
    byte order of the host system, use `sys.byteorder' as the byte order value.

    The signed keyword-only argument indicates whether two's complement is
    used to represent the integer.