如何矢量化单词列表？

Question

I am working on SMS data where I have a list of words in my one column of dataframe I want to train a classifier to predict it's type and subtype. How would I convert the words into numerical format as they are in a list.

Answer 1

The idea is to use as vocabulary all the words found in this column across instances, except that the least frequent words should be removed (to avoid overfitting). Then for every instance the column is represented as vector of boolean features, where the nth value represents the nth word in the vocabulary: 1 if it is in the list for this instance, 0 if not.

In python you can use CountVectorizer, considering every list in the column as a sentence.