How to convert an int to a list containing is binary representation fast in python?

Question

Currently I have the following version but it's my main bottleneck and it's quite slow.

def intToBinary(Input):
    bStrInput = format(Input, "016b")
    bStrInput = list(bStrInput)
    bInput = list(map(int, bStrInput))
    return bInput

any ideas how to speed up this code?

I'm using this in a Tensorflow project, for hot encoding conversion of integers. The function takes in 2-byte integer (in the range [0, 65536)) and outputs a list of integers with values 0 and 1:

>>> intToBinary(50411)
[1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1, 1]

The result is passed to a tensor with tInput = torch.tensor(bInput, dtype=torch.uint8) .

Answer 1

Your version, improved slightly

Your version can avoid a few opcodes by not using intermediary variables and a conversion to list:

def intToBinary(Input):
    return list(map(int, format(Input, "016b")))

Pure Python still, bitshifting option

However, you can make it faster by not converting to string then to integers again. If all you need is bits, then use bit manipulation:

def int_to_binary(v):
    return [(v >> i) & 1 for i in range(15, -1, -1)]

This shifts the bits of the input integer to the right by 15, 14, etc. steps, then masks out that shifted integer with 1 to get the bit value for the right-most bit each time.

Speed comparison, using 1000 random integers to reduce variance to acceptable levels:

>>> import sys, platform, psutil
>>> sys.version_info
sys.version_info(major=3, minor=7, micro=0, releaselevel='final', serial=0)
>>> platform.platform(), psutil.cpu_freq().current / 1000, psutil.cpu_count(), psutil.virtual_memory().total // (1024 ** 3)
('Darwin-17.7.0-x86_64-i386-64bit', 2.9, 8, 16)
>>> from timeit import Timer
>>> from random import randrange
>>> testvalues = [randrange(2**16) for _ in range(1000)]
>>> count, total = Timer("for i in tv: t(i)", "from __main__ import intToBinary as t, testvalues as tv").autorange()
>>> (total / count) * (10 ** 3)   # milliseconds
3.2812212200224167
>>> count, total = Timer("for i in tv: t(i)", "from __main__ import int_to_binary as t, testvalues as tv").autorange()
>>> (total / count) * (10 ** 3)   # milliseconds
2.2861225200176705

So int_to_binary() is about 1.5 times as fast, about 2.3 milliseconds to produce 1000 results, versus just over 3.3 for the optimised string manipulation version.

The base loop and function call takes 7.4 microseconds on my machine:

>>> count, total = Timer("for i in tv: pass", "from __main__ import testvalues as tv; t = lambda i: None").autorange()
>>> (total / count) * (10 ** 3)
0.007374252940062434

so the base per-call timings are about 3.27 microseconds vs 2.28 microseconds for the bit-manipulation version.

What can Numpy do

If you are using Tensorflow, you'll also have numpy operations available, which can convert uint8 to binary using the numpy.unpackbits() function ; uint16 needs to be 'viewed' as uint8 first:

import numpy as np

def int_to_bits_np(v):
    return np.unpackbits(np.array([v], dtype=np.uint16).view(np.uint8)).tolist()

This converts to numpy array, back to a list of Python integers again, so is not that efficient on just one value:

>>> count, total = Timer("for i in tv: t(i)", "from __main__ import int_to_bits_np as t, testvalues as tv").autorange()
>>> (total / count) * (10 ** 3)
2.654717969999183

Faster than your version, slower than bitshifting.

Numpy vectorised option

You probably want to not convert back to a list, since the numpy array already has the right dtype for your tensor here. You would also use this on a large number of values ; such as the whole 1000 integers in the input:

def int_to_bits_array(varray):
    """Convert an array of uint16 values to binary"""
    return np.unpackbits(varray.reshape(varray.shape[0], 1).view(np.uint8), axis=1)

which is way, way, way faster:

>>> testvalues_array = np.array(testvalues, dtype=np.uint16)
>>> int_to_bits_array(testvalues_array)
array([[1, 1, 0, ..., 1, 1, 0],
       [0, 1, 1, ..., 1, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       ...,
       [1, 1, 1, ..., 0, 1, 0],
       [0, 0, 0, ..., 1, 1, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=uint8)
>>> count, total = Timer("t(tva)", "from __main__ import int_to_bits_array as t, testvalues_array as tva").autorange()
>>> (total / count) * (10 ** 3)  # milliseconds
0.007919690339913358
>>> (total / count) * (10 ** 6)  # microseconds
7.919690339913359

Yes, that's 1000 values converted to binary in one step, processing all values in 8 microseconds. This scales up linearly to larger numbers; 1 million random values are converted in under 8 milliseconds:

>>> million_testvalues_array = np.random.randint(2 ** 16, size=10 ** 6, dtype=np.uint16)
>>> count, total = Timer("t(tva)", "from __main__ import int_to_bits_array as t, million_testvalues_array as tva").autorange()
>>> (total / count) * (10 ** 3)  # milliseconds
7.9162722200271665