CountVectorizer（）。适合scikit-learn Python给出内存错误

Question

I am working on an 8 class classification problem, Training set contains around 400,000 labeled entities, I am using CountVectorizer.fit() to vectorize the data, but I am getting a Memory error, I tried using HashingVectorizer instead, but in vain.

path = 'data/products.tsv' products = pd.read_table(path , header= None , names = ['label' , 'entry'])   
X = products.entry
y = products.label
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

# Vectorizing the Dataset
vect = CountVectorizer()
vect.fit(X_train.values.astype('U'))
X_train_dtm = vect.transform(X_train)
X_test_dtm = vect.transform(X_test)

Answer 1

You can set max_features which limit the memory usage of your vocabulary. The right value really depends on the task so you should treat it as a hyper parameter and try and tune it. In NLP (english) people usually use ~10,000 as a vocabulary size. You can also do the same with HashVectorizer but you risk hash collusions which will cause multiple words to increase the same counter.

path = 'data/products.tsv' products = pd.read_table(path , header= None , names = ['label' , 'entry'])   
X = products.entry
y = products.label
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

# Vectorizing the Dataset
vect = CountVectorizer(max_features=10000)
vect.fit(X_train.values.astype('U'))
X_train_dtm = vect.transform(X_train)
X_test_dtm = vect.transform(X_test)