我如何优化此代码以应对更大的数据集？

Question

I have a set of images with the same size. And I want to insert them into a dataframe, with the rows being the names of the images and the columns being the pixels. They are all in the same directory.

• I can already do this for a folder with a few images (as shown in the "Example for 7 images" link below), but when I try it for a dataset with 9912 images, the compile shows "killed". How can I optimize this code to get all the images?

from matplotlib import image
import numpy as np
import pandas as pd
import glob


columns = ["file"]
for i in range (150528):
    columns.append("pixel" + str(i))

df = pd.DataFrame(columns = columns)
i = 0
for file in glob.glob('/home/nuno/resizepics/*.jpg'): 
    imgarr = image.imread(file)
    imgarr = imgarr.flatten()

    df.loc[i,"file"] = file
    for j in range(len(imgarr)):
        df.iloc[i,j+1] = imgarr[j]

    i += 1

#print(df)


df.to_csv('pixels.csv')

Example for 7 images

Answer 1

If "killed" means it raises an error you can try using exeptions (try, except, else) and make it try again from the spot it stopped. You can also try to delay it a bit with time module because it works with large data.