Convert np.ndarray of tuples (dtype=object) into array with dtype=int

Question

I need to convert np arrays (short) of tuples to np arrays of ints.

The most obvious method doesn't work:

# array_of_tuples is given, this is just an example:
array_of_tuples = np.zeros(2, dtype=object)
array_of_tuples[0] = 1,2
array_of_tuples[1] = 2,3

np.array(array_of_tuples, dtype=int)

ValueError: setting an array element with a sequence.

Answer 1

It looks like placing the tuples into a pre-allocated buffer of fixed size and dtype is the way to go. It seems to avoid a lot of the overhead associated with computing sizes, raggedness and dtype.

Here are some slower alternatives and a benchmark:

• You can cheat and create a dtype with the requisite number of fields, since numpy supports conversion of tuples to custom dtypes:

   dt = np.dtype([('', int) for _ in range(len(array_of_tuples[0]))]) res = np.empty((len(array_of_tuples), len(array_of_tuples[0])), int) res.view(dt).ravel()[:] = array_of_tuples

• You can stack the array:

   np.stack(array_of_tuples, axis=0)

  Unfortunately, this is even slower than the other proposed methods.

• Pre-allocation does not help much:

   res = np.empty((len(array_of_tuples), len(array_of_tuples[0])), int) np.stack(array_of_tuples, out=res, axis=0)

• Trying to cheat using np.concatenate , which allows you to specify the output dtype does not help much either:

   np.concatenate(array_of_tuples, dtype=int).reshape(len(array_of_tuples), len(array_of_tuples[0]))

• And neither does pre-allocating the array:

   res = np.empty((len(array_of_tuples), len(array_of_tuples[0])), int) np.concatenate(array_of_tuples, out=res.ravel())

• You can also try to do the concatenation in python space, which is slow too:

   np.array(sum(array_of_tuples, start=()), dtype=int).reshape(len(array_of_tuples), len(array_of_tuples[0]))

  OR

   np.reshape(np.sum(array_of_tuples), (len(array_of_tuples), len(array_of_tuples[0])))

array_of_tuples = np.empty(100, dtype=object)
for i in range(len(array_of_tuples)):
    array_of_tuples[i] = tuple(range(i, i + 100))

%%timeit
res = np.empty((len(array_of_tuples), len(array_of_tuples[0])), int)
for i, res[i] in enumerate(array_of_tuples):
    pass
305 µs ± 8.55 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

dt = np.dtype([('', 'int',) for _ in range(100)])
%%timeit
res = np.empty((100, 100), int)
res.view(dt).ravel()[:] = array_of_tuples
334 µs ± 5.59 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit np.array(array_of_tuples.tolist())
478 µs ± 12.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
res = np.empty((100, 100), int)
np.concatenate(array_of_tuples, out=res.ravel())
500 µs ± 2.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit np.concatenate(array_of_tuples, dtype=int).reshape(100, 100)
504 µs ± 7.72 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
res = np.empty((100, 100), int)
np.stack(array_of_tuples, out=res, axis=0)
557 µs ± 25.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit np.stack(array_of_tuples, axis=0)
577 µs ± 6.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit np.array(sum(array_of_tuples, start=()), dtype=int).reshape(100, 100)
1.06 ms ± 11.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit np.reshape(np.sum(array_of_tuples), (100, 100))
1.26 ms ± 24.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)