Keras 和 Scikit-learn 的加权精度度量之间的差异

Question

Intro

Hej everyone,

I am working on my diploma thesis and I face a binary classification problem with imbalanced class contribution. I have around 10 times more negative ("0") labels as positive ("1") labels. For that reason I considered not only observing accuracy and ROC-AUC, but also weighted/ balanced accuracy and Precision-Recall-AUC.

I already asked the question on GitHub ( https://github.com/keras-team/keras/issues/12991 ) but the issue has not been answered yet so I thought this platform here might be the better place!

Issue description

During some calculations on the validation set in a custom callback I noticed, more or less by coincidence, that the weighted accuracy is always different from my results using sklearn.metrics.accuracy_score() .

Using Keras, weighted accuracy has to be declared in model.compile() and is a key in the logs{} dictionary after every epoch (and is also written to the log file by the CSVLogger callback or to the history object) or is returned as value in a list by model.evaluate() ,

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'], 
              weighted_metrics=['accuracy'])

I calculate the val_sample_weights vector based on the class contribution of the training set with the Sklearn.metrics function class_weight.compute_sample_weight() and with the help of class_weight.compute_class_weight() .

cls_weights = class_weight.compute_class_weight('balanced', np.unique(y_train._values), 
                                                y_train._values)
cls_weight_dict = {0: cls_weights[0], 1: cls_weights[1]}
val_sample_weights = class_weight.compute_sample_weight(cls_weight_dict, y_test._values)

In model.fit() I pass this vector togehter with the validation data and to sklearn.metrics.accuracy_score() I pass it to the parameter name sample_weight to compare the results on the same basis.

model_output = model.fit(x_train, y_train, epochs=500, batch_size=32, verbose=1,
                         validation_data=(x_test, y_test, val_sample_weights))

Furthermore, I derived the equation how Scitkit-learn computes the weighted accuracy from several easy examples and it seems that it's computed by the following equation (which seems quite reasonable to me):

LaTeX equation

TP, TN, FP and FN are the values reported in the confusion matrix and w_p and w_n are the class weights of the positive and negative class respectively.

An easy example to test it can be found here:

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html

Just for the sake of completeness, sklearn.metrics.accuracy_score(..., sample_weight=) returns the same result as sklearn.metrics.balanced_accuracy_score() .

System Information

• GeForce RTX 2080 Ti
• Keras 2.2.4
• Tensorflow-gpu 1.13.1
• Sklearn 0.19.2
• Python 3.6.8
• CUDA Version 10.0.130

Code example

I searched an easy example to make the issue easy to reproduce, even if the class imbalance here is weaker (1:2 not 1:10). It's based on the introductory tutorial to Keras which can be found here:

https://towardsdatascience.com/k-as-in-keras-simple-classification-model-a9d2d23d5b5a

The Pima Indianas onset diabets dataset will be downloaded, as done in the link above, from the repository of Jason Brownlee, the maker of the homepage Machine Learning Mastery. But I guess it can also be downloaded from various other sites.

So finally here's the code:

from keras.layers import Dense, Dropout
from keras.models import Sequential
from keras.regularizers import l2
import pandas as pd
import numpy as np
from sklearn.utils import class_weight
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

file = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/' \
       'pima-indians-diabetes.data.csv'

# Load csv data from file to data using pandas
data = pd.read_csv(file, names=['pregnancies', 'glucose', 'diastolic', 'triceps', 'insulin',
                                'bmi', 'dpf', 'age', 'diabetes'])

# Process data
data.head()
x = data.drop(columns=['diabetes'])
y = data['diabetes']

# Split into train and test
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.1, random_state=0)

# define a sequential model
model = Sequential()
# 1st hidden layer
model.add(Dense(100, activation='relu', input_dim=8, kernel_regularizer=l2(0.01)))
model.add(Dropout(0.3))
# 2nd hidden layer
model.add(Dense(100, activation='relu', kernel_regularizer=l2(0.01)))
model.add(Dropout(0.3))
# Output layer
model.add(Dense(1, activation='sigmoid'))
# Compilation with weighted metrics
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'], 
                         weighted_metrics=['accuracy'])

# Calculate validation _sample_weights_ based on the class distribution of train labels and 
# apply it to test labels using Sklearn
cls_weights = class_weight.compute_class_weight('balanced', np.unique(y_train._values), 
                                                y_train._values)
cls_weight_dict = {0: cls_weights[0], 1: cls_weights[1]}
val_sample_weights = class_weight.compute_sample_weight(cls_weight_dict, y_test._values)

# Train model
model_output = model.fit(x_train, y_train, epochs=500, batch_size=32, verbose=1,
                         validation_data=(x_test, y_test, val_sample_weights))

# Predict model
y_pred = model.predict(x_test, batch_size=32, verbose=1)

# Classify predictions based on threshold at 0.5
y_pred_binary = (y_pred > 0.5) * 1

# Sklearn metrics
sklearn_accuracy = accuracy_score(y_test, y_pred_binary)
sklearn_weighted_accuracy = accuracy_score(y_test, y_pred_binary, 
                                           sample_weight=val_sample_weights)

# metric_list has 3 entries: [0] val_loss weighted by val_sample_weights, [1] val_accuracy 
# [2] val_weighted_accuracy
metric_list = model.evaluate(x_test, y_test, batch_size=32, verbose=1, 
                             sample_weight=val_sample_weights)

print('sklearn_accuracy=%.3f' %sklearn_accuracy)
print('sklearn_weighted_accuracy=%.3f' %sklearn_weighted_accuracy)
print('keras_evaluate_accuracy=%.3f' %metric_list[1])
print('keras_evaluate_weighted_accuracy=%.3f' %metric_list[2])

Results and summary

For example I get:

sklearn_accuracy=0.792

sklearn_weighted_accuracy=0.718

keras_evaluate_accuracy=0.792

keras_evaluate_weighted_accuracy=0.712

The "unweighted" accuracy value is the same, both for Sklearn as for Keras. The difference isn't really big, but it grows bigger as the dataset becomes more imbalanced. For example for my task it always differs around 5% from each other!

Maybe I'm missing something and it's supposed to be like that, but anyways it's confusing that Keras and Sklearn provide different values, especially thinking of the whole class_weights and sample_weights thing as a topic hard to get into. Unfortunately I'm not too deep into Keras to search in the Keras code on my own.

I would really appreciate receiving any answers!

Answer 1

I repeated your exact toy example and actually found that sklearn and keras do give the same results. I repeated the experiment 5 times to ensure it wasn't by chance and indeed the results were identical each time. For one of the runs for example:

sklearn_accuracy=0.831
sklearn_weighted_accuracy=0.800
keras_evaluate_accuracy=0.831
keras_evaluate_weighted_accuracy=0.800

FYI I'm using sklearn and keras versions:

0.20.3
2.3.1

respectively. See this google colab example: https://colab.research.google.com/drive/1b5pqbp9TXfKiY0ucEIngvz6_Tc4mo_QX