tf-idf vectorizer for multi-label classification problem

Question

I have a multi-label classification project for a large number of texts. I used the tf-Idf vectorizer on the texts (train_v['doc_text']) as follows:

tfidf_transformer = TfidfTransformer()
X_counts = count_vect.fit_transform(train_v['doc_text']) 
X_tfidf = tfidf_transformer.fit_transform(X_counts) 
x_train_tfidf, x_test_tfidf, y_train_tfidf, y_test_tfidf = train_test_split(X_tfidf_r, label_vs, test_size=0.33, random_state=9000)
sgd = SGDClassifier(loss='hinge', penalty='l2', random_state=42, max_iter=25, tol=None, fit_intercept=True, alpha = 0.000009  )

now, I need to use the same vectorizer on a set of features (test_v['doc_text'])to predict the labels. however, when I use the following

X_counts_test = count_vect.fit_transform(test_v['doc_text']) 
X_tfidf_test = tfidf_transformer.fit_transform(X_counts_test) 
predictions_test = clf.predict(X_tfidf_test)

I get an error message

ValueError: X has 388894 features per sample; expecting 330204

any idea on how to deal with this?

Thanks.

Answer 1

The problem is you are using fit_transform here which make the TfidfTransform() fit on the test data and then transform it.

Rather use transform method on it.

Also, you should use TfidfVectorizer

In my opinion the code should be:

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_transformer = TfidfVectorizer()
# X_counts = count_vect.fit_transform(train_v['doc_text']) 
X_tfidf = tfidf_transformer.fit_transform(train_v['doc_text']) 
x_train_tfidf, x_test_tfidf, y_train_tfidf, y_test_tfidf = train_test_split(X_tfidf, label_vs, test_size=0.33, random_state=9000)
sgd = SGDClassifier(loss='hinge', penalty='l2', random_state=42, max_iter=25, tol=None, fit_intercept=True, alpha = 0.000009  )

# X_counts_test = count_vect.fit_transform(test_v['doc_text']) 
X_tfidf_test = tfidf_transformer.transform(test_v['doc_text']) 
predictions_test = clf.predict(X_tfidf_test)

Also, why are you using count_vect I think it has no usability here and in train_test_split you are using X_tfidf_r which is not mentioned anywhere.