Numpy 连接中的 Memory 错误 (np.concatenate)

Question

I'm using TfidfVectorizer and for that, I need to concatenate my training and testing data, which gives me a memory error.

train_text = data['comment_text'].values.astype(str)
test_text = test_data['comment_text'].values.astype(str)
all_text = np.concatenate([train_text, test_text])

Above is the code piece and below is the error.

Traceback (most recent call last):
File "Model.py", line 68, in <module>
  EasyEnsembleClassfier(data,test_data)
File "Model.py", line 30, in EasyEnsembleClassfier
  all_text = np.concatenate([train_text, test_text])
MemoryError

Any suggestions on how to resolve this?

Answer 1

I think this is probably due to insufficient memory or the data size is very large. Can you try again by reducing the data size? You can also consider killing some unnecessary processes going on your machine (on windows -- you can do it by visiting control panel) to free up some memory. Let me know if it works.