MemoryError：无法为形状为 (323313, 3435) 且数据类型为 float64 的数组分配 8.27 GiB

Question

I have extension(example .exe,.py,.xml,.doc etc) table in my dataframe. after running on terminal I am getting above error on large data set.

encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit(features['Extension'].values.reshape(-1, 1))
temp = encoder.transform(features['Extension'].values.reshape(-1, 1)).toarray()  #GETTING ERROR on this
print("Size of array in bytes",getsizeof(temp))
print("Array :-",temp)
print("Shape :- ",features.shape, temp.shape)
features.drop(columns=['Extension'], axis=1, inplace=True)
dump(encoder, os.path.join(os.getcwd(), 'model_dumps', 'encoder.pkl'))
features.drop(columns=['Extension'], axis=1, inplace=True)
features = featureScaling(features)
features = np.concatenate((features, temp), axis=1)

OUTPUT -

1) Size of array in bytes :- 8884558912
2) Array :- 
[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [1. 0. 0. ... 0. 0. 0.]
 [1. 0. 0. ... 0. 0. 0.]
 [1. 0. 0. ... 0. 0. 0.]]
3)Shape :- (323310, 8) (323310, 3435)

Answer 1

that's funny.

MemoryError: Unable to allocate 8.27 GiB for an array with shape (323313, 3435) and data type float64

most modern computers don't have more than 8 Gb of RAM. Looks like you have 8 and python is not able to fit all this data in the memory. Try buying another computer with more ram or upgrade your existing one. This will definitely fix the issue.