dynamically append N-dimensional array

Question

If each array has the shape (1000, 2, 100), it is easy to use

con = np.concatenate((array_A, array_B))

to concatenate them, thus con has the shape (2000, 2, 100).

I want to dynamically append or concatenate "con" in a function. The step is described as following:

First, read data from the first file and process data to generate an array.

Secondly, read date from the second file and append generated array into the first array

....

def arrayappend():
     for i in range(n):
         #read data from file_0 to file_n-1
         data = read(file_i)
         #data processing to generate an array with shape (1000, 2, 100)
         con = function(data)
         # append con

Answer 1

Assuming all your files produce the same shape objects and you want to join them on the 1st dimension, there are several options:

 alist = []
 for f in files:
    data = foo(f)
    alist.append(f)
 arr = np.concatenate(alist, axis=0)

concatenate takes a list. There are variations if you want to add a new axis ( np.array(alist) , np.stack etc).

Append to a list is fast, since it just means adding a pointer to the data object. concatenate creates a new array from the components; it's compiled but still relatively slower.

If you must/want to make a new array at each stage you could write:

 arr = function(files[0])
 for f in files[1:]:
     data = foo(f)
     arr = np.concatenate((arr, data), axis=0)

This probably is slower, though, if the file loading step is slow enough you might not notice a difference.

With care you might be able start with arr = np.zeros((0,2,100)) and read all files in the loop. You have to make sure the initial 'empty' array has a compatible shape. New users often have problems with this.

Answer 2

If you absolutely want to do it during iteration then:

def arrayappend():
  con = None
  for i, d in enumerate(files_list):
    data = function(d)
    con = data if i is 0 else np.vstack([con, data])

This should stack it vertically.

Answer 3

Very non pretty, but does it achieve what you want? It is way unoptimized.

def arrayappend():
    for i in range(n):
        data = read(file_i)
        try:
            con
            con = np.concatenate((con, function(data)))
        except NameError:
            con = function(data)
    return con

First loop will take the except branch, subsequent wont.