如何在Python中提高磁盘读取速度

Question

I use Python for Image analysis. The first step in my code is to load the images from disk to a big 20GB uint8 array. This step is taking a very long time, loading about 10MB/s, and the cpu is idling during the task.

This seems extremely slow. Am I making an obvious mistake? How can I improve performance? Is it a problem with the numpy array type?

# find all image files in working folder
FileNames = []      # FileNames is a list of image names
workingFolder = 'C:/folder'
for (dirpath, dirnames, filenames) in os.walk(workingFolder):
     FileNames.extend(filenames)
FileNames.sort() # Sorted by image number 
 imNumber = len(FileNames) # Number of Images

 # AllImages initialize
 img = Image.open(workingFolder+'/'+FileNames[0])
 AllImages = np.zeros((img.size[0],img.size[1], imNumber),dtype=np.uint8)

 for ii in range(imNumber):
     img = Image.open(workingFolder+'/'+FileNames[ii])
     AllImages[:,:,ii] = img

Thanks a lot for your help.

Answer 1

Since the CPU is idling it sounds that it's the disk that is the bottle neck. 10 Mb/s is somewhat slow, but not that slow that it reminds me of stone age hard disks. If it were numpy I'd expect the CPU to be busy running numpy code rather than being idle.

Note that there maybe two ways the CPU will be waiting for the disk. First of course you will need to read the data from disk, but also since the data is 20GB the data may be big enough to require it to be swapped to disk. The normal solution to this type of situation is to memory map the file (which will avoid moving data from disk to swap).

Try to check if you can read the files faster by other means. For example on linux you could use dd if=/path/to/image of=/tmp/output bs=8k count=10k; rm -f /tmp/output dd if=/path/to/image of=/tmp/output bs=8k count=10k; rm -f /tmp/output to check the speed of read to ram. See this question for more information on checking disk performance.