Faster way to modify a numpy array

Question

I have a large numpy array with 4 million rows and 4 columns (shape = (4000000,4))

I need to modify/ decrease the number of rows, based on the value in fourth column. For example few of my rows in my data set look like the following:

a = np.array([[1.32, 24.42, 224.21312, 0],[1.32, 24.42, 224.21312, 0],[1.32, 24.42, 224.21312, 1],[1.32, 24.42, 224.21312, 1],[1.32, 24.42, 224.21312, 0]]);

My result should be the following (only rows with last column value = 1)

b = [1.32, 24.42, 224.21312, 1],[1.32, 24.42, 224.21312, 1]

A for loop to go through each row is taking a long time to process.

I have 200 of these arrays, so I am already using multiprocessing for each array.

Looking for suggestions.

Answer 1

does this work for you?

a[a[:,3] == 1]

gives:

array([[  1.32   ,  24.42   , 224.21312,   1.     ],
       [  1.32   ,  24.42   , 224.21312,   1.     ]])

Answer 2

You can convert it to dataframe and operate your operations there and then convert back to array:

df = pd.DataFrame(a)
df = df[df[3] == 1]
a = df.as_matrix()

Output:

array([[  1.32   ,  24.42   , 224.21312,   1.     ],
       [  1.32   ,  24.42   , 224.21312,   1.     ]])