仅当满足每行元素上的条件时，才计算2D数组特定列的均值和方差

Question

I have a 2D numpy array with dimension (690L, 15L). I need to compute a columns wise mean on this dataset only in some particolar columns, but with a condition: I need to include a row if and only if an element in the same row at specific column satisfy a condition. Let's me more cleare with some code.

f = open("data.data")
dataset =  np.loadtxt(fname = f, delimiter = ',')

I have array with fullfilled with indexes where I need to perform mean (and variance)

index_catego = [0, 3, 4, 5, 7, 8, 10, 11]

The condition is that the dataset[i, 14] == 1 As output I want an 1D array with length like len(index_catego) where each element of this array is the mean of the previously columns

output = [mean_of_index_0, mean_of_index_3, ..., mean_of_index_11]

I am using Python recently but I am sure there is a cool way of doing this with np.where , mask , np.mean or something else.

I already implement a solution, but it is not elegant and I am not sure if it is correct.

import numpy as np

index_catego = [0, 3, 4, 5, 7, 8, 10, 11]

matrix_mean_positive = []
matrix_variance_positive = []
matrix_mean_negative = []
matrix_variance_negative = []

n_positive = 0
n_negative = 0

sum_positive = np.empty(len(index_catego))
sum_negative = np.empty(len(index_catego))


for i in range(dataset.shape[0]):
    if dataset[i, 14] == 0:
        n_positive = n_positive + 1
        j = 0
        for k in index_catego:
            sum_positive[j] = sum_positive[j] + dataset[i, k]
            j = j + 1
    else:
        n_negative = n_negative + 1
        j = 0
        for k in index_catego:
            sum_negative[j] = sum_negative[j] + dataset[i, k]
            j = j + 1

for item in np.nditer(sum_positive):
    matrix_mean_positive.append(item / n_positive)

for item in np.nditer(sum_negative):
    matrix_mean_negative.append(item / n_negative)

print(matrix_mean_positive)
print(matrix_mean_negative)

If you wanna try your solution, I put some data example

1,22.08,11.46,2,4,4,1.585,0,0,0,1,2,100,1213,0
0,22.67,7,2,8,4,0.165,0,0,0,0,2,160,1,0
0,29.58,1.75,1,4,4,1.25,0,0,0,1,2,280,1,0
0,21.67,11.5,1,5,3,0,1,1,11,1,2,0,1,1
1,20.17,8.17,2,6,4,1.96,1,1,14,0,2,60,159,1
0,15.83,0.585,2,8,8,1.5,1,1,2,0,2,100,1,1
1,17.42,6.5,2,3,4,0.125,0,0,0,0,2,60,101,0

Thanks for you help.

UPDATE 1: I tried with this

output_positive = dataset[:, index_catego][dataset[:, 14] == 0]
mean_p = output_positive.mean(axis = 0)
print(mean_p)

output_negative = dataset[:, index_catego][dataset[:, 14] == 1]
mean_n = output_negative.mean(axis = 0)
print(mean_n)

but means computed by the first (solution not cool) and the second solution (one line cool solotion) are all different. I checked what dataset[:, index_catego][dataset[:, 14] == 0] and dataset[:, index_catego][dataset[:, 14] == 1] select and seems correct (right dimension and right element).

UPDATE 2: Ok, the first solution is wrong because (for example) the first column have as element only 0 and 1, but as mean return a value > 1. I do not know where I failed. Seems that the positive class is correct (or at least plausible), while negative class are not even plausible.

So, is it second solution correct? Is there a better way of doing it?

UPDATE 3: I think I found the problem with the first solution: I am using jupyter notebook and sometimes (not all the times) when I rerun the same cell where the first solution is, element in matrix_mean_positive and matrix_mean_negative are doubled. If someone know why, could be point me?

Now both solution return the same means.

Answer 1

在重新运行之前在Jupyter Notebook中执行内核->重新启动以清除内存