Efficient way to extract individual bit values from a hexa string

Question

I'm developing a real-time application where I have to process a line of data as fast as possible to send it over to an app. These lines arrive at a very fast rate, around 40k per minute. The task is to extract the value of certain individual bits from the hexa data in the line. I have a solution already but I doubt it's the most efficient one, so I'm asking if you can improve it.

A sample line of data:

p1                p2      p3     len  data
1497383697        0120    000    5    00 30 63 4f 15

len is how many bytes are in the data, data is what we're working with. Let's say I want to extract 3 bits starting from the 11th from the left. Converting the hexa to binary with padding:
0x0030634f15 = 0000 0000 00 11 0 000 0110 0011 0100 1111 0001 0101
The wanted value is 0b110 which is 6 in decimal.

My working solution for the problem is this:

# 11 and 3 in the example
start = config.getint(p, 'start') 
length = config.getint(p, 'length')

parts = line.split()
hexadata = ''.join(parts[4:])
bindata = bin(int(hexadata, 16))[2:].zfill(len(hexadata) * 4)
val = int(bindata[start:start + length], 2)

val will hold the value 6 in the end. Any other, more efficent way to do this? Thank you

Answer 1

Instead of using string operations, it's faster to convert the input to a number and use bit operations:

parts = line.split(maxsplit=4)

# remove spaces in the number and convert it to int from base 16
num = int(parts[4].replace(' ', ''), 16)

# create a bit mask with exactly `length` 1s
mask = (1 << length) - 1

# calculate the offset from the right
shift = 40 - start - length

# shift the value to the right and apply the binary mask to get our value
val = (num >> shift) & mask

---

According to my timings, the bit operations are faster by about 20%. Timing results with 1 million iterations:

string_ops  2.735653492003621 seconds
bit_ops     2.190693126998667 seconds