Create a 3D matrix from a generator of 2D matrices

Question

I am using pandas to read a number of .csv files. Each file will produce a 3x5 dataframe.

dtx = (pd.read_csv(f).values for f in get_filelist(datadirectory))

I want to combine all the data from the various files in order to create a 3 dimensional array (if I have 10 files then I want to end up with an array of shape: 10x3x5 )

I could create an empty python list and append all the arrays found in dtx using a for loop, but I would like a more pythonic solution. I have tried

np.concatenate([tf for tf in dtx])

without having the desired effect. How can I concatenate all of the data I read from my .csv files into a big 3D array?

Answer 1

You can add a new dimension to the arrays and concatenate them:

dtx = (pd.read_csv(f).values for f in get_filelist(datadirectory))
np.concatenate([tf[np.newaxis] for tf in dtx], axis=0)

Example:

np.concatenate([tf[np.newaxis] for tf in (np.arange(4).reshape((2,2)) for i in range(3))], axis=0)
>> array([[[0, 1],
        [2, 3]],

       [[0, 1],
        [2, 3]],

       [[0, 1],
        [2, 3]]])