重新创建稀疏矩阵列以进行ML模型预测

Question

I have been working on a model using sklearn and a big part of it utilizes the CountVectorizer() function to create a sparse matrix from a set of strings in the training set.

ex:

vectorizer = CountVectorizer(max_features=3000)
sparse_matrix = vectorizer.fit_transform(corpus).toarray()

After exporting the model, whats the best way to format the data I would like to create a prediction on to match the feature names created by the training? Should I be exporting (via pickle?) vectorizer.get_feature_names() as well and then use that? Or is there a better way?

In other words, if in my training set vectorizer.get_feature_names() = ['apple', 'dog', 'cat'] and I would like to make a prediction on 'hello cat' , what should my method for feature extraction on the prediction request be? Correct me if Im wrong, but result of the feature extraction would need to be [0, 0, 1] to match the model.

I could also be totally off on my approach here as well, so any help or suggestions are appreciated

Thanks!

Answer 1

When you type

vectorizer = CountVectorizer(max_features=3000)
sparse_matrix = vectorizer.fit_transform(corpus).toarray()

This vectorizer is used to fit the VOCABULARY of words that you have in corpus

so, use the SAME vectorizer to transform another dataset and you will see the frequency of words of your new dataset CORRESPONDING to the vocabulary of corpus

Remember you perform fit_transform(X) to say "Use the vocabulary of X ", and you do that just once. And then you do JUST tranform(Y) as a way of saying, "Whatever you used for X, use them as columns, and fit the terms in Y into these X columns

from sklearn.feature_extraction.text import CountVectorizer
corpus = ['love dogs, hate cows, and also pigs, actually dogs too']
vectorizer = CountVectorizer(max_features=3000)
sparse_matrix = vectorizer.fit_transform(corpus)
df = pd.DataFrame(sparse_matrix.toarray())
df.columns = vectorizer.get_feature_names()
print(df)

Would give you this:

   actually  also  and  cows  dogs  hate  love  pigs  too
0         1     1    1     1     2     1     1     1    1

and then:

test = vectorizer.transform(['hello cat']) #Notice how I use transform and not fit_transform
df = pd.DataFrame(test.toarray())
df.columns = vectorizer.get_feature_names()
print(df)

   actually  also  and  cat  cows  dogs  hate  love  pigs  too
0         0     0    0    1     0     0     0     0     0    0

Notice how hello cat was fit on the vocabulary of terms that fit_transform was called on. So in your feature extraction, you FIT your 'hello cat' to the vocabulary you called fit_transform on!

And now, USE ALL THESE 10 columns as FEATURES to predict a label y . What you are doing is called a Vector Space Model