正在加载对于 RAM 来说太大的.npy 文件

Question

I am trying to load a large.npy file (~800 MB) into Google Collab, but every time I try to do so, the Collab instance crashes due to RAM over usage.

     import numpy as np
     a=np.load('oddata.npy',allow_pickle=True)

I am using the basic Collab instance with 12 GB RAM.

I have tried using mmap, but it returns this error: ValueError: Array can't be memory-mapped: Python objects in dtype.

Is there anyway around the problem, like breaking the.npy file into chunks or converting it into another file format.

Best,

Araf

Answer 1

I think there is more to your problem than insufficient memory. The 12 GB allocated in your Collab instance should be more than enough to read a 800MB file. To confirm, I ran a simple test on my Raspberry Pi (which only has 4GB RAM). It can create a 1GB.npy file and read it back in to a new array. Here is the code:

import numpy as np
nimg, n0, n1 = 1000, 512, 512
arr = np.arange(nimg*n0*n1).reshape(nimg,n0,n1)
print(arr.dtype, arr.shape)

np.save('SO_67671598.npy',arr)
arr2 = np.load('SO_67671598.npy')    
print(arr2.dtype, arr2.shape)

I get the same result with or without the allow_pickle=True parameter. Note that allow_pickle=True is not recommended (for security reasons). It is necessary when you are loading object arrays. I suggest you run this test in your Collab instance, and see what you get.