How can I get accuracy from confusion matrix in tensorflow or Keras in the form of a tensor?

Question

I want to get UAR (unweighted accuracy) from confusion matrix to monitor UAR of validation data. However, it is difficult to deal with tensor.

https://www.davidtvs.com/keras-custom-metrics/

I did refer to this site and try to create my own metrics in Keras. I am making the metrics by using the first method to use both ModelCheckpoint and EarlyStopping supported by Keras.

model.compile(loss='categorical_crossentropy',optimizer=adam, metrics=['accuracy', uar_accuracy])

However, I don't know how to define the uar_accuracy function.

            def uar_accuracy(y_true, y_pred):

            # Calculate the label from one-hot encoding
            pred_class_label = K.argmax(y_pred, axis=-1)
            true_class_label = K.argmax(y_true, axis=-1)


            cf_mat = tf.confusion_matrix(true_class_label, pred_class_label )

            diag = tf.linalg.tensor_diag_part(cf_mat)
            uar = K.mean(diag)

            return uar

This result returns the average of the number of data right for each class. But I do not want the average of the number of correct data, but I want the average of the correct probabilities for each class.

How can I implement it?

I have implemented the following for the numpy type, not the Tensor type using sklearn.metrics and collections library

            def get_accuracy_and_cnf_matrix(label, predict):
            uar = 0
            accuracy = []
            cnf_matrix = confusion_matrix(label, predict)
            diag=np.diagonal(cnf_matrix)
            for index,i in enumerate(diag):
                uar+=i/collections.Counter(label)[index]

            # cnf_marix (Number of corrects -> Accuracy)    
            cnf_matrix = np.transpose(cnf_matrix)
            cnf_matrix = cnf_matrix*100 / cnf_matrix.astype(np.int).sum(axis=0)
            cnf_matrix = np.transpose(cnf_matrix).astype(float)
            cnf_matrix = np.around(cnf_matrix, decimals=2)   

            # WAR, UAR
            test_weighted_accuracy = np.sum(label==predict)/len(label)*100
            test_unweighted_accuracy = uar/len(cnf_matrix)*100    
            accuracy.append(test_weighted_accuracy)
            accuracy.append(test_unweighted_accuracy)

            return np.around(np.array(accuracy),decimals=2), cnf_matrix  

Answer 1

You can use tf.reduce_sum to compute the sum of each row in your confusion matrix. This corresponds to the total number of data points for each class. Then you divide the diagonal elements with this row sum to compute the ratio of correctly predicted examples per class.

def non_nan_average(x):
    # Computes the average of all elements that are not NaN in a rank 1 tensor
    nan_mask = tf.debugging.is_nan(x)
    x = tf.boolean_mask(x, tf.logical_not(nan_mask))
    return K.mean(x)


def uar_accuracy(y_true, y_pred):
    # Calculate the label from one-hot encoding
    pred_class_label = K.argmax(y_pred, axis=-1)
    true_class_label = K.argmax(y_true, axis=-1)

    cf_mat = tf.confusion_matrix(true_class_label, pred_class_label )

    diag = tf.linalg.tensor_diag_part(cf_mat)    

    # Calculate the total number of data examples for each class
    total_per_class = tf.reduce_sum(cf_mat, axis=1)

    acc_per_class = diag / tf.maximum(1, total_per_class)  
    uar = non_nan_average(acc_per_class)

    return uar