Python-预处理大量图像数据-循环冻结

Question

I have the following code in my Python Script:

def process_data(data):
    features = []

    for i in range(int(len(data))): # Should iterate 33'000 times (length of data)
        for j in range(3):
            img = plt.imread(data[i][j].strip()) # Data is from a csv file
            img = normalize(img)  # Simple calculation

            lis = img.flatten().tolist()
            features += lis

return features

This should run for about 33'000*3 times to preprocess the whole data. However, after around 10'000-12'000 iterations, the script slows down radically, freezes and sometimes my machine (i7 - 3.6GHz, 8GB RAM) freezes as well.

What can I do? It's difficult to split up the data. I was told to use the Keras' fit_generator before but how would I do that?

Answer 1

Depending on how large those image files are, you could be running out of RAM and being forced to swap. That would cause your system to be sluggish. Instead of gathering all the processed files into list, can you write them out one at a time?