Appending a pair of numpy arrays in a specific format to a 3rd array

Question

I'm extracting data points in pairs from a data set. A pair consists of 2 numpy arrays, each shaped (3, 30, 30) . Let's call them X1 and Y1 . The next pair will then be X2 and Y2 , followed by X3 and Y3 , etc. I don't know how many pairs there will be in total, so I have to use something like np.append .

So I want something like:

>>X1, Y1 = extract_X_and_Y_from_data(data)
>>pair1 = np.array([X1, Y1])
>>pair1.shape
(2, 3, 30, 30)
>>list_of_pairs.some_append_function(pair1)
>>list_of_pairs.shape
(1, 2, 3, 30, 30)

>>X2, Y2 = extract_X_and_Y_from_data(data)
>>pair2 = np.array([X2, Y2])
>>list_of_pairs.some_append_function(pair2)
>>list_of_pairs.shape
(2, 2, 3, 30, 30)

...

>>X50, Y50 = extract_X_and_Y_from_data(data)
>>pair50 = np.array([X50, Y50])
>>list_of_pairs.some_append_function(pair50)
>>list_of_pairs.shape
(50, 2, 3, 30, 30)

All in all, I need the final list_of_pairs to be a numpy array of shape (no_of_pairs, 2, 3, 30, 30) . np.append keeps giving me (no_of_pairs, 2) , and I'm not too sure why.

Note: np.concatenate , vstack or hstack are tricky to implement because they can't seem to execute the first instance, ie appending the first pair to an initially empty list_of_pairs .

Thanks!

Answer 1

With list append

list_of_pairs = []   # real list
for data in database:
    X1, Y1 = extract_X_and_Y_from_data(data)
    pair1 = np.array([X1, Y1])
    list_of_pairs.some_append_function(pair1)
array5d = np.array(list_of_pairs)
>> array5d.shape
(50, 2, 3, 30, 30)

appending to a list is relatively fast since it just adds a pointer to the list. Your pair array remains in memory.

np.array(alist) builds a new array, joining the components on a new dimension (same as in np.array([[1,2,3],[4,5,6]]) )

There is a new function np.stack that gives you a more control over which dimension is new. All the stack functions end up calling np.concatenate . That includes the misnamed (and oft misused) np.append . concatenate requires matching dimensions (in the joining direction). The various stacks just adjust the overall number of dimensions.