在多维数组上使用numpy.where

Question

I have a 2D array, each row represents an output of a classifier that classifies some input to 3 categories (array size is 1000 * 3 ) :

0.3 0.3 0.3
0.3 0.3 1.0
1.0 0.3 0.3
0.3 0.3 0.3
0.3 1.0 0.3
...

I want to get a list of all the inputs that the classifier is "not sure" about them. And I'm defining "not sure" as no category is above 0.8.

To solve it I use :

np.where(model1_preds.max(axis=1) < 0.8)

This works great.

But now I have 6 classifiers (that have analyzed the same inputs in the same order), and an array 6 * 1000 * 3 representing their results.

I want to find 2 things:

1. All the inputs that at least one classifier was "not sure" about.
2. All the inputs that all the classifier were "not sure" about.

I assume the general direction is something like this :

np.stack(np.where(model_preds.max(axis=1) < 0.8) for model_preds in all_preds)

But it won't work because python don't know what I mean in the for loop.

Answer 1

Alternatively to np.where :

res_all_unsure = preds[:,np.amax(preds, axis=(0,2)) <= 0.8,:]
res_one_unsure = preds[:,preds.max(-1).min(0) <= 0.8,:]

Answer 2

If it is already a 6×1000×3 matrix preds , you can first np.transpose() it into a 1000×6×3 matrix.

y = preds.transpose(1,0,2)  # preds is the input matrix, 6x1000x3

Next we can turn it into a 1000×6 matrix where for each experiment and for each classifier, we know whether all the values were less than 0.8 by stating:

y = np.all(y<0.8,axis=2)

Finally we can use another np.all() to verify where all the classifiers were unsure:

all_classifiers_unsure = np.where(np.all(y,axis=1))  # all classifiers

Or where any of the classifiers was unsure:

any_classifier_unsure = np.where(np.any(y,axis=1))   # any of the classifiers

We can write it shorter like:

experiment_classifier = np.all(preds.transpose(1,0,2) < 0.8,axis=2)
all_classifiers_unsure = np.where(np.all(experiment_classifier,axis=1))
any_classifier_unsure = np.where(np.any(experiment_classifier,axis=1))

Although I am quite confident, please validate by checking a few indices (ones that are true and ones that are not true).

EDIT

You can still use your proposed method of .max() < 0.8 , but with axis=2 :

experiment_classifier = preds.transpose(1,0,2)
all_classifiers_unsure = np.where(np.all(experiment_classifier,axis=1))
any_classifier_unsure = np.where(np.any(experiment_classifier,axis=1))