how to unpack every n bits from a byte array into an integer array

Question

data comes from a binary file, and I convert it to a byte_array using numpy as below:

byte_array = np.frombuffer(data, dtype=np.uint8)

Is there any convenient way to extract every n bits (8<=n<=20) from this byte_array then cast those n bits to an integer and store all the integers in an array?

The only way that I can think of is to go through the byte_array one by one, extract a full byte first, if the total bits extracted so far is less than n, extract the remaining bits, then concatenate with the first byte and form an output integer, store that integer to the output array.

For example, n = 12, byte_array[0:3] = 123, 99, 100

output_integer[0] = 123 | ((99&0xF)<<8) # bits 0..7 from 123 and bits 8..11 from the lower 4 bits of 99 
output_integer[1] = (99>>4) | (100<<4) # bits 0..3 from higher 4 bits of 99 and bits 4..11 from 100

Answer 1

You can create a generator for yielding the bits and use for example more_itertools.chunked to process the bit stream in n-chunks:

from more_itertools import chunked

def get_bits(data):
    for b in data:
        yield from f'{bin(b)[2:]:0>8}'

data = b'0123456789'
n = 20
for bits in chunked(get_bits(data), n):
    print(int(''.join(bits), base=2))

In the above example, if (r := len(data)*8 % n) != 0 then the last bits will be of length r < n . If that is undesired, you can either pad the bit stream beforehand on the left (ie at the start), which requires knowledge of its length, or use for example more_itertools.grouper to pad it on the right (ie at the end).