Python-在不使用所有可用内存的情况下加载大量图像

Question

I have about 1.5 GB of images that I need to process. The problem is that when I try loading them as np arrays I seem to use up all of my ram (8 GB).

Here is my method for loading images:

def load_image( infilename ) :
    img = Image.open( infilename )
    img.load()
    data = np.asarray( img, dtype="int32" )
    img.close()
    del img
    return data

I thought closing and deleting the img would help, but it doesn't. Can this have something to do with garbage collection?

Code to loop through all images in a list of file names:

for i in range(len(files)):
    imgArray = imgs.load_image(files[i])
    images.append(imgArray)
    shapes.append(np.shape(imgArray))

Is there a better way?

Answer 1

It might be worth it to load the image files one by one using PIL to get their size tuples, collect your statistics about averages and what not, then open them again in numpy or PIL to do the actual processing. You might also want to consider sampling for the statistics part so you don't need to load all of them, not that it should take that long anyway, PIL is relatively efficient.

Answer 2

You may be able to use manual garbage collection to clear some of the memory between loops:

def memclear():
    import gc   #garbage collector
    cleared = gc.collect()

    print(cleared)

call: memclear() at the end of each loop, so:

for i in range(len(files)):
    imgArray = imgs.load_image(files[i])
    images.append(imgArray)
    shapes.append(np.shape(imgArray))
    memclear()

Hopefully this fixes it. I'm assuming this was downvoted because it manually calls garbage cleaning, which is generally frowned upon, but unfortunately it seems to be necessary sometimes.