numpy array containing multi-dimension numpy arrays with variable shape

Question

I have a list of numpy arrays, whose shape is one of the following: (10,4,4,20), (10,4,6,20) . I want to convert the list to a numpy array. Since, they are of different shapes, I can't just stack them. So, I thought of creating numpy array considering each array as an object, as in here . I tried the below:

b = numpy.array(a)
b = numpy.array(a, dtype=object)

where a is the list of numpy arrays. Both are giving me the following error:

ValueError: could not broadcast input array from shape (10,4,4,20) into shape (10,4)

How can I convert that list to numpy array?

Example :

import numpy
a = [numpy.random.random((10,4,4,20)),
     numpy.random.random((10,4,6,20)),
     numpy.random.random((10,4,6,20)),
     numpy.random.random((10,4,4,20)),
     numpy.random.random((10,4,6,20)),
     numpy.random.random((10,4,6,20)),
     numpy.random.random((10,4,4,20)),
     numpy.random.random((10,4,4,20)),
     numpy.random.random((10,4,6,20))
    ]
b = numpy.array(a)

Use Case :
I know numpy array of objects are not efficient, but I'm not doing any operations on them. Usually, I have a list of same shape numpy arrays and so I can easily stack them. This array is passed to another function, which selects certain elements only. If my data is numpy array, I can just do b[[1,3,8]] . But I can't do the same with list. I get the following error if I try the same with list

c = a[[1,3,8]]
TypeError: list indices must be integers or slices, not list

Answer 1

np.array(alist) will make an object dtype array if the list arrays differ in the first dimension. But in your case they differ in the 3rd, producing this error. In effect, it can't unambiguously determine where the containing dimension ends, and where the objects begin.

In [270]: alist = [np.ones((10,4,4,20),int), np.zeros((10,4,6,20),int)]                                
In [271]: arr = np.array(alist)                                                                        
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-271-3fd8e9bd05a9> in <module>
----> 1 arr = np.array(alist)

ValueError: could not broadcast input array from shape (10,4,4,20) into shape (10,4)

Instead we need to make an object array of the right size, and copy the list to it. Sometimes this copy still produces broadcasting errors, but here it seems to be ok:

In [272]: arr = np.empty(2, object)                                                                    
In [273]: arr                                                                                          
Out[273]: array([None, None], dtype=object)
In [274]: arr[:] = alist                                                                               
In [275]: arr                                                                                          
Out[275]: 
array([array([[[[1, 1, 1, ..., 1, 1, 1],
         [1, 1, 1, ..., 1, 1, 1],
         [1, 1, 1, ..., 1, 1, 1],
...
         [0, 0, 0, ..., 0, 0, 0],
         [0, 0, 0, ..., 0, 0, 0]]]])], dtype=object)
In [276]: arr[0].shape                                                                                 
Out[276]: (10, 4, 4, 20)
In [277]: arr[1].shape                                                                                 
Out[277]: (10, 4, 6, 20)