Copy 2D array to a 3D one - Python / NumPy

Question

I programmed a little bit when I was younger but I was never really good. I find Python perfect for what I want to do.

I have an Excel file which contains data (64 columns, 18496 lines) that I read using the numpy genfromtxt function. I want to put everything in a 3D matrix named H. I use three loops to do so but I know that this is not the most efficient.

data = np.genfromtxt(filename, delimiter = ";",skiprows = 11) 
H = np.zeros((N,N,Nt))

for k in np.arange(N):
    for l in np.arange(N):            
        for m in np.arange(Nt):    
            H[k,l,m] = data[m+Nt*k,l]

Is there a cleaver (faster computing wise) to do so. I though about using numpy shape but I'm not able to do it.

Thanks

Answer 1

You could reshape with np.reshape & then re-arrange dimensions with np.transpose , like so -

H = data.reshape(N,Nt,N).transpose(0,2,1)

Instead of np.transpose , one can also use np.swapaxes as basically we are swapping axes 1,2 there, like so -

H = data.reshape(N,Nt,N).swapaxes(1,2)

Sample run -

In [300]: N = 2
     ...: Nt = 3
     ...: data = np.random.randint(0,9,(N*Nt,N))
     ...: 

In [301]: data
Out[301]: 
array([[3, 6],
       [7, 4],
       [8, 1],
       [8, 7],
       [4, 8],
       [2, 3]])

In [302]: H = np.zeros((N,N,Nt),dtype=data.dtype)
     ...: for k in np.arange(N):
     ...:     for l in np.arange(N):            
     ...:         for m in np.arange(Nt):    
     ...:             H[k,l,m] = data[m+Nt*k,l]
     ...:             

In [303]: H
Out[303]: 
array([[[3, 7, 8],
        [6, 4, 1]],

       [[8, 4, 2],
        [7, 8, 3]]])

In [304]: data.reshape(N,Nt,N).transpose(0,2,1)
Out[304]: 
array([[[3, 7, 8],
        [6, 4, 1]],

       [[8, 4, 2],
        [7, 8, 3]]])

Runtime test -

In [8]: # Input
   ...: N = 10
   ...: Nt = 10*50
   ...: data = np.random.randint(0,9,(N*Nt,N))
   ...: 
   ...: def original_app(data):
   ...:     H = np.zeros((N,N,Nt),dtype=data.dtype)
   ...:     for k in np.arange(N):
   ...:         for l in np.arange(N):            
   ...:             for m in np.arange(Nt):    
   ...:                 H[k,l,m] = data[m+Nt*k,l]
   ...:     return H
   ...: 

In [9]: np.allclose(original_app(data),data.reshape(N,Nt,N).transpose(0,2,1))
Out[9]: True

In [10]: %timeit original_app(data)
10 loops, best of 3: 56.1 ms per loop

In [11]: %timeit data.reshape(N,Nt,N).transpose(0,2,1)
1000000 loops, best of 3: 1.25 µs per loop