如何解释和研究不平衡数据中的完美准确性，精确度，召回率，F1和AUC（我不信任）

Question

I have a largely imbalanced multi-labeled dataset.

Something unexpected came up in the results. As expected, using logistic regression classifier, labels having higher frequency achieved reasonable f1-score and auc-score (ie: 0.6-0.7), and those labels with less than 10% representation in the data expectedly got 0 for f-1 and 0.5 for auc-score.

But when I run the same thing with SVC and Naive Bayes classifiers, some of these lower-frequency labels (for example: out of the 7000 samples, a minor class may have 10 samples) showed 100% accuracy, f-1, precision, recall, and auc-score, which I don't understand. I don't trust these perfect results given such low training sample available. I also tried different random seed to split the training and test sets, and received the same results.

Classifiers

Logistic regression classifier
Pipeline(memory=None,
     steps=[('tfidf', TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), norm='l2', preprocessor=None, smooth_idf=True,
 ..._state=None, solver='sag', tol=0.0001,
          verbose=0, warm_start=False),
          n_jobs=1))])

Naive Bayes classifier
Pipeline(memory=None,
     steps=[('tfidf', TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), norm='l2', preprocessor=None, smooth_idf=True,
 ...assifier(estimator=MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True),
          n_jobs=1))])

SVC classifier
Pipeline(memory=None,
     steps=[('tfidf', TfidfVectorizer(analyzer='word', binary=False, decode_error='strict',
        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
        lowercase=True, max_df=1.0, max_features=None, min_df=1,
        ngram_range=(1, 1), norm='l2', preprocessor=None, smooth_idf=True,
 ...lti_class='ovr', penalty='l2', random_state=None, tol=0.0001,
     verbose=0),
          n_jobs=1))])

Answer 1

To me, your result seems at least credible. Logistic regression tends toward a median characterization of the data, finding a single equation to characterize the differences among classes. Given the non-trivial quantity of data, it looks for the least error fit for that equation.

SVC and Bayes are much more sensitive to discernible boundaries, even when far from the "mainstream of data. Those algorithms work more on the "us against the world" (aka "one versus all") view of each class. Thus, it doesn't surprise me that they can find a reasonable way to discriminate between a set of ten elements and "everything else".

Can you find a useful visualization tool to display the boundaries found by each method? If not, can you at least visualize the data set, with observations color-coded? If you can see a distinct separation for a set of ten points, then I would expect SVC or Naive Bayes to find something comparable.

Answer 2

Did you check on how many samples these metrics were calculated? If there were, eg only two samples for testing 100% is not that odd, given the low number of testing samples.

Additionally, since you have imbalanced data did you consider measures like the balanced accuracy or Mathews correlation coefficient (MCC) to gain insight into the predictive performance? Models can have a very high AUC while disregarding the minority class completely. If this also coincides with eg only majority class samples in the test set that can also lead to these unexpected results.