MemoryError: 无法分配形状为 (2515, 406272) 且数据类型为 float32 的数组

Question

I am trying to make a finger knuckle recognition using CNN on Spyder on Anaconda. I have 2515 images that i am dealing with and it seems 8 GB is not enough to do this. What can i do? Should i get another stick of RAM or making this project on Kaggle/Colab will be a better approach?

arr = np.array(finger_dat)
arr = arr.reshape((2515, 406272))

arr = arr / 255

#############Making dataset####################

label = np.array(range(503))
label = np.repeat(label, 5)

dataset = pd.DataFrame(arr)
dataset['label'] = label

X = dataset.iloc[:, 0:406272]
y = dataset.iloc[:, -1]

from sklearn.tree import DecisionTreeClassifier
dtf = DecisionTreeClassifier()
dtf.fit(X, y)

dtf.score(X, y)

Answer 1

Try to process the data in batches.

Have a look in this doc. from sklearn: Strategies to scale computationally: bigger data

Answer 2

In your place I would try in this order:

colab, change code to load batches change code to use database to feed train buy more ram but you are probably going to need > 40Gb.