Scikit学习返回错误的分类报告和准确性得分

Question

I'm training an SVM on 1200 examples of label 2 and 1200 examples of label 1 with an RBF kernel. I thought I was getting 77% accuracy, and I was getting accuracy using sklearn.metrics.accuracy_score . But when I hand-rolled my own precision score, like so:

def naive_accuracy(true, pred):
    number_correct = 0
    i = 0
    for y in true:
        if pred[i] == y:
            number_correct += 1.0
    return number_correct / len(true)

It got 50%. I believe I've wasted weeks of work based on a false accuracy score and classification report. Can anyone supply me with an explanation for why this has happened? I'm very, very confused as to how this could have happened. I don't see what I'm doing wrong. And when I tested the metrics.accuracy_score function on some dummy data like pred = [1, 1, 2, 2]; test = [1, 2, 1, 2] , and it gave me 50% like you'd expect. I think accuracy_score might be erring due to my specific data somehow.

I have 27-feature vectors and 1200 vectors of class 1 and 1200 vectors of class 2. My code is the following:

X = scale(np.asarray(X))
y = np.asarray(y)
X_train, X_test, y_train, y_test = train_test_split(X, y)

######## SVM ########
clf = svm.SVC()
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
# 77%
print "SVM Accuracy:", accuracy_score(y_test, y_pred) # debugging
# 50%
print "*True* SVM Accuracy:", naive_accuracy(y_test, y_pred) # in-house debugging
# also 77%!
print "Classification report:\n", classification_report(y_test, y_pred) # debugging

Answer 1

Your implementation of naive_score is buggy. You are comparing the first element with all the others ( i is never updated).

I would've just left a comment if not for the test case you've designed, which prevented you from zeroing in on the bug yourself.

Try running your code with:

pred = list([1, 2, 2, 2]); 
test = list([1, 1, 1, 1])

The accuracy returned will be 1.0 !

Also worth noting is the fact that if the classes are uniformly distributed, then the expected accuracy returned by the buggy code can be shown to be 50% on any random test set.

It is also a good idea to have a test suite with several test cases. A single test case can rarely test all the possible scenarios in non trivial cases.

Though not really needed, here is what you should do instead:

def naive_accuracy(true, pred):
    number_correct = 0
    i = 0
    for i, y in enumerate(true):
        if pred[i] == y:
            number_correct += 1.0
    return number_correct / len(true)