Updating a NumPy array by adding columns

Question

I am working with a large dataset and I would like to make a new array by adding columns, updating the array by opening a new file, taking a piece from it and adding this to my new array.

I have already tried the following code:

import numpy as np
Powers = np.array([])
with open('paths powers.tex', 'r') as paths_list:
    for file_path in paths_list:
        with open(file_path.strip(), 'r') as file:
            data = np.loadtxt(file_path.strip())
            Pname = data[0:32446,0]
            Powers = np.append(Powers,Pname, axis = 1)
            np.savetxt("Powers.txt", Powers)

However, what it does here is just adding the stuff from Pname in the bottom of the array, making a large 1D array instead of adding new columns and making an ndarray .

I have also tried this with numpy.insert , numpy.hstack and numpy.concatenate and I tried changing the shape of Pname . Unfortunately, they all give me the same result.

Answer 1

Have you tried numpy.column_stack ?

Powers = np.column_stack([Powers,Pname])

However, the array is empty first, so make sure that the array isn't empty before concatenating or you will get a dimension mismatch error:

import numpy as np
Powers = np.array([])
with open('paths powers.tex', 'r') as paths_list:
    for file_path in paths_list:
        with open(file_path.strip(), 'r') as file:
            data = np.loadtxt(file_path.strip())
            Pname = data[0:32446,0]
            if len(Powers) == 0:
                Powers = Pname[:,None]
            else:
                Powers = np.column_stack([Powers,Pname])
            np.savetxt("Powers.txt", Powers)

len(Powers) will check the amount of rows that exist in Powers . At the start, this should be 0 so at the first iteration, this is true and we will need to explicitly make Powers equal to a one column 2D array that consists of the first column in your file. Powers = Pname[:,None] will help you do this, which is the same as Powers = Pname[:,np.newaxis] . This transforms a 1D array into a 2D array with a singleton column. Now, the problem is that when you have 1D arrays in numpy , they are agnostic of whether they are rows or columns. Therefore, you must explicitly convert the arrays into columns before appending. numpy.column_stack takes care of that for you.

However, you'll also need to make sure that the Powers is a 2D matrix with one column the first time the loop iterates. Should you not want to use numpy.column_stack , you can still certainly use numpy.append , but make sure that what you're concatenating to the array is a column. The thing we talked about above should help you do this:

import numpy as np
Powers = np.array([])
with open('paths powers.tex', 'r') as paths_list:
    for file_path in paths_list:
        with open(file_path.strip(), 'r') as file:
            data = np.loadtxt(file_path.strip())
            Pname = data[0:32446,0]
            if len(Powers) == 0:
                Powers = Pname[:,None]
            else:      
                Pname = Pname[:,None]
                Powers = np.append(Powers, Pname, axis=1) 
            np.savetxt("Powers.txt", Powers)

The second statement ensures that the array becomes a 2D array with a singleton column before concatenating.