Create ndarray from list with mismatched axis>0 size

Question

I want to save a list of Numpy arrays to a file. The list is of the following shape:

my_list = [np.ones((2, 515, 3)), np.ones((2, 853, 3))]

However, when I try to save it using np.savez , the list tries to get converted into an Numpy array. Doing np.array(my_list, dtype='object') gives the error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-6fcbd172df30> in <module>()
----> 1 np.array([np.ones((2, 515, 3)), np.ones((2, 853, 3))], dtype='object')

ValueError: could not broadcast input array from shape (2,515,3) into shape (2)

However, if the axis=0 dimension is mismatched instead of the axis=1 dimension, such as my_list = [np.ones((515, 3)), np.ones((853, 3))] , I no longer get this error.

Why does the mistmatched axis dimension affect the ability to Numpy array from objects?

Although there are work-arounds possible to break up the array into a save-able format, I'm mostly interested about why the conversion failure is happening and how to get around it.

Answer 1

In [77]: my_list = [np.ones((2, 515, 3)), np.ones((2, 853, 3))]

Save with the *args parameter, or with a **kwargs dictionary

In [78]: np.savez('test',*my_list)
In [79]: ll = np.load('test.npz')
In [80]: list(ll.keys())
Out[80]: ['arr_0', 'arr_1']
In [81]: ll['arr_0'].shape
Out[81]: (2, 515, 3)
In [82]: ll['arr_1'].shape
Out[82]: (2, 853, 3)

or with named keywarods/dictionary

In [85]: np.savez('test',x=my_list[0],y=my_list[1])

np.savez('test', my_list) , first turns my_list into an array - or tries to

In [83]: np.array(my_list)
...
ValueError: could not broadcast input array from shape (2,515,3) into shape (2)

When trying to create an array from a list of arrays there are 3 possible outcomes: a higher dimensional array (if dimensions match), an object array (if dimensions don't match), or this error (if the dimensions sort-of match).

The object dtype case:

In [86]: arr=np.array([np.ones((515, 3)), np.ones((853, 3))])
In [87]: arr.shape
Out[87]: (2,)
In [88]: arr.dtype
Out[88]: dtype('O')

The surest way to create an object array is to preallocate it

In [90]: arr = np.zeros((2,), object)
In [91]: arr[...]=my_list

The shape of arr has to match the nesting of the sublists/arrays in my_list , otherwise you'll get broadcasting errors. arr can be reshaped after loading.