如何在scikit中使用SVM学习分类不同的测试数据以检查垃圾邮件

Question

I am doing review spam detection using SVM in scikit learn. for this task i am using gold standard data set of truthful and deceptive reviews of each 400. Now i have done so far is to train and test split of this same dataset and find accuracy.

Now I want to train my SVM classifier using this dataset and then want to classify my new downloaded test data different then original data set.

How can I do this task. My code so far is:

def main():
        init();
        dir_path ='C:\spam\hotel-reviews'
        files = sklearn.datasets.load_files(dir_path)
        model = CountVectorizer()
        X_train = model.fit_transform(files.data)

       tf_transformer = sklearn.feature_extraction.text.TfidfTransformer(use_idf=True).fit(word_counts)
        X = tf_transformer.transform(word_counts)
        #print X
        print '\n\n'

    # create classifier
        clf = sklearn.svm.LinearSVC()
    # test the classifier
        test_classifier(X, files.target, clf, test_size=0.2, y_names=files.target_names, confusion=False)

def test_classifier(X, y, clf, test_size=0.3, y_names=None, confusion=False):
    #train-test split
    X_train, X_test, y_train, y_test = sklearn.cross_validation.train_test_split(X, y, test_size=test_size)

    clf.fit(X_train, y_train)
    y_predicted = clf.predict(X_test)

    print sklearn.metrics.classification_report(y_test, y_predicted, target_names=y_names)

if __name__ == '__main__':
    main()

Now i want to classify my own different review data of 500 reviews in reviews.txt file using above trained classifier, so how can i do this?

Answer 1

To score your data two steps are needed. Either return clf and usea separate method for scoring or you can use within same method. This is the workflow

def scoreData(clf): 
    x_for_predict = loadScoringData("reviews.txt") # Signature only. assuming same data format without target variable 
    y_predict = clf.predict(x_for_predict)
    plotResults(clf, y_predict)# just a signature.