如何将TfidfVectorizer的输出馈送到Sklearn中的LinearSVC分类器？

Question

I'm trying to build a linear classifier using LinearSVC in Scikit learn. I decided to use the tf-idf vectorization for the purpose of vectorizing the text input. The code I wrote is:

review_corpus = list(train_data_df['text'])
vectorizer = TfidfVectorizer(max_df = 0.9,stop_words = 'english')
%timeit tfidf_matrix = vectorizer.fit_transform(review_corpus)

I now want to train an SVM model using this tfidf_matrix and use it to predict the class/label for the corresponding test set: test_data_df['text'] . The problem(s) I'm having:

1. Is it correct to use only the training data to build the TfIdfVectorizer or should I use both the training and testing text data to build the vectorizer?

2. The main issue is: How do I get the matrix representation for the testing data? Currently, I'm not sure how to get the tfidf score from the vectorizer for the different documents in the test set. What I tried was to loop through the Pandas series test_data_df['text'] and then do:

    tfidf_matrix.todense(list(text) 

for each text in the Series, put the result into a list and finally make a numpy array out of it but I get a Memory Error.

Answer 1

1. You should use only the training data to build the TfIdfVectorizer() . This will ensure that you are not leaking any information about the test data during training process.

2. Use

    tfidf_matrix_test = vectorizer.transform(test_data_df['text']) 

Now you can feed the tfidf_matrix_test to the classifier.

PS:

Try to avoid casting the sparse_matrix output of the Vectorizer to list or dense array. Because it is memory intensive and classifier will also take more computation time while training/prediction.