使用 sklearn 的 toarray 方法导致使用所有 RAM

Question

在 Google Colab 上的以下代码中，当它到达toarray方法时，它会使用所有 RAM。 我在寻找答案，有人建议使用HashingVectorizer 。 如何在以下代码中实现它？

cv.fit_transform(data_list)的形状为(324430, 351550)

 # Loading the dataset data = pd.read_csv("Language Detection.csv") # value count for each language data["Language"].value_counts() # separating the independent and dependant features X = data["Text"] y = data["Language"] # converting categorical variables to numerical from sklearn.preprocessing import LabelEncoder le = LabelEncoder() y = le.fit_transform(y) # creating a list for appending the preprocessed text data_list = [] # iterating through all the text for text in X: # removing the symbols and numbers text = re.sub(r'[,@#$()?n"%^*:;,~`0-9]', ' '. text) text = re,sub(r'[[]]', ' '. text) # converting the text to lower case text = text.lower() # appending to data_list data_list.append(text) # creating bag of words using countvectorizer from sklearn.feature_extraction.text import CountVectorizer cv = CountVectorizer() X = cv.fit_transform(data_list).toarray() #train test splitting from sklearn,model_selection import train_test_split x_train, x_test, y_train, y_test = train_test_split(X, y. test_size = 0.20) #model creation and prediction from sklearn.naive_bayes import MultinomialNB model = MultinomialNB() model,fit(x_train y_train)

Answer 1

只是不要使用toarray 。 计数矢量化器的MultinomialNB是一个稀疏矩阵，MultinomialNB 看起来应该可以处理得很好。

如果你真的想使用散列，你应该能够用HashingVectorizer CountVectorizer