多级分类的ROC曲线，在python中没有一个对所有

Question

I have a multi-class classification problem with 9 different classes. I am using the AdaBoostClassifier class from scikit-learn to train my model without using the one vs all technique, as the number of classes is very high and it might be inefficient.

I have tried using the tips from the documentation in scikit learn [1], but there the one vs all technique is used, which is substantially different. In my approach I only get one prediction per event, ie if I have n classes, the outcome of the prediction is a single value within the n classes. For the one vs all approach, on the other hand, the outcome of the prediction is an array of size n with a sort of likelihood value per class.

[1] https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#sphx-glr-auto-examples-model-selection-plot-roc-py

The code is:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt # Matplotlib plotting library for basic visualisation
%matplotlib inline

from sklearn.model_selection import train_test_split 
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_curve, auc
from sklearn import preprocessing


# Read data
df = pd.read_pickle('data.pkl')

# Create the dependent variable class
# This will substitute each of the n classes from 
# text to number
factor = pd.factorize(df['target_var'])
df.target_var= factor[0]
definitions = factor[1]

X = df.drop('target_var', axis=1)
y = df['target_var]

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

bdt_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=2),
    n_estimators=250,
    learning_rate=0.3)

bdt_clf.fit(X_train, y_train)

y_pred = bdt_clf.predict(X_test)

#Reverse factorize (converting y_pred from 0s,1s, 2s, etc. to their original values
reversefactor = dict(zip(range(9),definitions))
y_test_rev = np.vectorize(reversefactor.get)(y_test)
y_pred_rev = np.vectorize(reversefactor.get)(y_pred)

I tried directly with the roc curve function, and also binarising the labels, but I always get the same error message.

def multiclass_roc_auc(y_test, y_pred):
    lb = preprocessing.LabelBinarizer()
    lb.fit(y_test)
    y_test = lb.transform(y_test)
    y_pred = lb.transform(y_pred)
    return roc_curve(y_test, y_pred)

multiclass_roc_auc(y_test, y_pred_test)

The error message is:

ValueError: multilabel-indicator format is not supported

How could this be sorted out? Am I missing some important concept?

Answer 1

An ROC (receiver operating characteristic curve) is formed from a plot of true positives and false positives from a binary classifier.

The area under the curve gives an indication of accuracy for a binary classifier.

For multiclass problems, you can find the accuracy, but this can be misleading if your data tends to fall into the categories non-uniformly. Appropriate sampling can overcome this.

The AdaBoostClassifier you are using will give you a score , showing the mean accuracy.