preallocation of numpy array of numpy arrays

Question

I read about how important it is to preallocate a numpy array. In my case I am, however, not sure how to do this. I want to preallocate an nxm matrix. That sounds simple enough

M = np.zeros((n,m))

However, what if my matrix is a matrix of matrices? So what if each of these nxm elements is actually of the form

np.array([[t], [x0,x1,x2], [y0,y1,y2]])

I know that in that case, M would have the shape (n,m,3). As an example, later I want to have something like this

[[[[0], [0,1,2], [3,4,5]],
    [[1], [10,11,12], [13,14,15]]], 
[[[0], [100,101,102], [103,104,105]],
    [[1], [110,111,112], [113,114,115]]]]

I tried simply doing

M = np.zeros((2,2,3))

but then

M[0,0,:] = np.array([[0], [0,1,2], [3,4,5]])

will give me an error

ValueError: setting an array element with a sequence.

Can I not preallocate this monster? Or should I approach this in a completely different way?

Thanks for your help

Answer 1

If you know you will only store values t, y, x for each point in n,m then it may be easier, and faster computationally, to have three numpy arrays.

So:

M_T = np.zeros((n,m))
M_Y = np.zeros((n,m))
M_X = np.zeros((n,m))

I believe you can now type 'normal' python operators to do array logic, such as:

MX = np.ones((n,m))
MY = np.ones((n,m))
MT = MX + MY
MT ** MT
_ * 7.5

By defining array-friendly functions (similarly to MATLAB) you will get a big speed increase for calculations.

Of course if you need more variables at each point then this may become unwieldy.

Answer 2

You have to make sure you preallocate the correct number of dimensions and elements along each dimension to use simple assignments to fill it.

For example you want to save 3 2x3 matrices:

number_of_matrices = 3
matrix_dim_1 = 2
matrix_dim_2 = 3

M = np.empty((number_of_matrices, matrix_dim_1, matrix_dim_2))
M[0] = np.array([[  0,   1,   2], [  3,   4,   5]])
M[1] = np.array([[100, 101, 102], [103, 104, 105]])
M[2] = np.array([[ 10,  11,  12], [ 13,  14,  15]])

M
#array([[[   0.,    1.,    2.],           # matrix 1
#        [   3.,    4.,    5.]],
# 
#        [[ 100.,  101.,  102.],          # matrix 2
#        [ 103.,  104.,  105.]],
#
#       [[  10.,   11.,   12.],           # matrix 3
#        [  13.,   14.,   15.]]])

---

You're approach contains some problems. The array you want to save is not a valid ndimensional numpy array:

np.array([[0], [0,1,2], [3,4,5]])
# array([[0], [0, 1, 2], [3, 4, 5]], dtype=object)
#                                    |----!!----|
#         ^-------^----------^       3 items in first dimension
#         ^                          1 item in first item of 2nd dim
#              ^--^--^               3 items in second item of 2nd dim
#                         ^--^--^    3 items in third item of 2nd dim    

It just creates an 3 item array containing python list objects. You probably want to have an array containing numbers so you need to care about dimensions. Your np.array([[0], [0,1,2], [3,4,5]]) could be a 3x1 array or a 3x3 array, numpy doesn't know what to do in this case and saves it as objects (the array now has only 1 dimension!).

---

The other problem is that you want to set one element of the preallocated array with another array that contains more than one element. This is not possible (except you already have an object -array). You have two options here:

1. Fill as many elements in the preallocated array as are required by the array:

    M[0, :, :] = np.array([[0,1,2], [3,4,5]]) # ^--------------------^--------^ First dimension has 2 items # ^---------------^-^-^ Second dimension has 3 items # ^------------------------^-^-^ dito # if it's the first dimension you could also use M[0] 

2. Create a object array and set the element ( not recommended, you loose most of the advantages of numpy arrays ):

    M = np.empty((3), dtype='object') M[0] = np.array([[0,1,2], [3,4,5]]) M[1] = np.array([[0,1,2], [3,4,5]]) M[2] = np.array([[0,1,2], [3,4,5]]) M #array([array([[0, 1, 2], # [3, 4, 5]]), # array([[0, 1, 2], # [3, 4, 5]]), # array([[0, 1, 2], # [3, 4, 5]])], dtype=object)