getting roc_auc_score nan for cross validation of multiclass classification

Question

I'm working on a multiclass classification problem with imbalanced data with 3 classes, I used stratifiedkfolds to split data and SMOTE method to oversample it. when I use cross-validation to evaluate my models I get results for F1_score but for roc_auc I only get nan value

for key, classifier in classifiers.items():
    classifier.fit(X_sm, y_sm)
    training_score1 = cross_val_score(classifier, X_sm, y_sm,scoring=make_scorer(f1_score, average='macro', labels=[2]), cv=5)
    print("Classifiers: ", classifier.__class__.__name__, "Has a training score of", round(training_score1.mean(), 2) * 100, "% F1  score")
    training_score2 = cross_val_score(classifier, X_sm, y_sm,scoring=make_scorer(roc_auc_score, average='macro',multi_class='ovo'), cv=5)
    print("Classifiers: ", classifier.__class__.__name__, "Has a training score of", round(training_score2.mean(), 2) * 100, "% Roc_auc score")

X_sm and y_sm are both arrays and the results in this case are:

Classifiers:  LogisticRegression Has a training score of 77.0 % F1  score
Classifiers:  LogisticRegression Has a training score of nan % Roc_auc score
Classifiers:  KNeighborsClassifier Has a training score of 94.0 % F1  score
Classifiers:  KNeighborsClassifier Has a training score of nan % Roc_auc score
Classifiers:  SVC Has a training score of 89.0 % F1  score
Classifiers:  SVC Has a training score of nan % Roc_auc score
Classifiers:  DecisionTreeClassifier Has a training score of 83.0 % F1  score
Classifiers:  DecisionTreeClassifier Has a training score of nan % Roc_auc score

I tried to use cross_validate but it's not working for me.

Answer 1

The auROC metric requires a continuous confidence measure, as opposed to the hard class predictions, so you need to set needs_proba=True or needs_threshold=True . The latter uses the classifier's decision_function or predict_proba , whereas the first only tries to use predict_proba ; since SVMs are not natively probabilistic, you'll want needs_threshold . (Do not set either of these for F1, which only uses hard class predictions.)