将大量图像保存为数组

Question

I have a high number of videos and I want to extract the frames, pre-process them and then create an array for each video. So far I have created the arrays but the final size of each array is too big for all of the videos. I have 224 videos, each resulting in a 6GB array totaling more than 1.2TB. I have tried using numpy.save and pickle.dump but both create the same size on the system. Do you have a recommendation or an alternative way in general?

Answer 1

Do these steps for each of the videos:

1. Load the data into one NumPy array.
2. Write to disk using np.save() with the extension .npy .
3. Add the .npy file to a .zip compressed archive using the zipfile module.

The end result will be as if you loaded all 224 arrays and saved them at once using np.savez_compressed , but it will only use enough RAM to process a single video at a time, instead of having to store all the uncompressed data in memory at once.

Finally, np.load() (or zipfile ) can be used to load the data from disk, one video at a time, or even using concurrent.futures.ThreadPoolExecutor to load multiple files at once using multiple cores for decompression to save time (you can get speedup almost linear with the number of cores, if your disk is fast).