无论我的训练集有多小，测试准确度始终很高

Question

I am doing a project where I am trying to classify comments into various categories: "toxic","severe_toxic","obscene","insult","identity_hate". The dataset I am using is from this Kaggle challenge: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge . The current issue I am facing is that no matter how small a training dataset I fit my data on, when I predict labels for the test data, my accuracy is always around or above 90%. In this case I am training on 15 rows of data and testing on 159,556 rows. I would normally be excited to have a high testing accuracy, but in this case, I feel like I am doing something wrong.

I am reading the data into a pandas dataframe:

trainData = pd.read_csv('train.csv')

Here is what the data looks like when printed:

                      id                                       comment_text  \
0       0000997932d777bf  Explanation\nWhy the edits made under my usern...   
1       000103f0d9cfb60f  D'aww! He matches this background colour I'm s...   
2       000113f07ec002fd  Hey man, I'm really not trying to edit war. It...   
3       0001b41b1c6bb37e  "\nMore\nI can't make any real suggestions on ...   
4       0001d958c54c6e35  You, sir, are my hero. Any chance you remember...   
...                  ...                                                ...   
159566  ffe987279560d7ff  ":::::And for the second time of asking, when ...   
159567  ffea4adeee384e90  You should be ashamed of yourself \n\nThat is ...   
159568  ffee36eab5c267c9  Spitzer \n\nUmm, theres no actual article for ...   
159569  fff125370e4aaaf3  And it looks like it was actually you who put ...   
159570  fff46fc426af1f9a  "\nAnd ... I really don't think you understand...   

        toxic  severe_toxic  obscene  threat  insult  identity_hate  
0           0             0        0       0       0              0  
1           0             0        0       0       0              0  
2           0             0        0       0       0              0  
3           0             0        0       0       0              0  
4           0             0        0       0       0              0  
...       ...           ...      ...     ...     ...            ...  
159566      0             0        0       0       0              0  
159567      0             0        0       0       0              0  
159568      0             0        0       0       0              0  
159569      0             0        0       0       0              0  
159570      0             0        0       0       0              0  

[159571 rows x 8 columns]

Then I split the data into train and test, using train_test_split:

X = trainData.drop(labels= ['id','toxic','severe_toxic','obscene','threat','insult','identity_hate'],axis=1)
Y = trainData.drop(labels = ['id','comment_text'],axis=1)

trainX,testX,trainY,testY = train_test_split(X,Y,test_size=0.9999,random_state=99)

I am using sklearn's HashingVectorizer to convert the comments into numerical vectors for classifying:

def hashVec():
    trainComments=[]
    testComments=[]
    for index,row in trainX.iterrows():
        trainComments.append(row['comment_text'])
    for index,row in testX.iterrows():
        testComments.append(row['comment_text'])
    vectorizer = HashingVectorizer()
    trainSamples = vectorizer.transform(trainComments)
    testSamples = vectorizer.transform(testComments)
    return trainSamples,testSamples

I am using OneVsRestClassifier and LogisticRegression from sklearn to fit and predict data for each of the 6 classes

def logRegOVR(trainSamples,testSamples):
    commentTypes=['toxic','severe_toxic','obscene','threat','insult','identity_hate']
    clf = OneVsRestClassifier(LogisticRegression(solver='sag'))
    for cType in commentTypes:
        print(cType,":")
        clf.fit(trainSamples,trainY[cType])
        pred1 = clf.predict(trainSamples)
        print("\tTrain Accuracy:",accuracy_score(trainY[cType],pred1))
        prediction = clf.predict(testSamples)
        print("\tTest Accuracy:",accuracy_score(testY[cType],prediction))

Finally, here is where I call the functions, and the output I get:

sol = hashVec()
logRegOVR(sol[0],sol[1])

toxic :
    Train Accuracy: 0.8666666666666667
    Test Accuracy: 0.9041590413397177
severe_toxic :
    Train Accuracy: 1.0
    Test Accuracy: 0.9900035097395272
obscene :
    Train Accuracy: 1.0
    Test Accuracy: 0.9470468048835519
threat :
    Train Accuracy: 1.0
    Test Accuracy: 0.9970041866178646
insult :
    Train Accuracy: 1.0
    Test Accuracy: 0.9506317531148938
identity_hate :
    Train Accuracy: 1.0
    Test Accuracy: 0.9911943142219659

The testing accuracy is very similar when I have a more reasonable train_test_split of 80% training and 20% testing.

Thank you for the assistance

Answer 1

You are not using a good metric : accuracy is not a good way to determine if you're doing right. I recommend you to look at what we call F1 score which is a mix between precision and recall and I found it more relevant to evaluate how my classifier is working

Answer 2

If it is an unbalanced dataset, accuracy does not mean a thing. If 90% of your dataset comments do not fall into any of those 'toxic' categories, and the model always predicts that a comment is 'clean', you still have your 90% accuracy.