Python加速硬盘上的随机磁盘读取

Question

i have a large set of files (each ~100kb) sitting on my HDD. For each step of my algorithm, i need to randomly select and read in about 1000 files. I use python and numpy.load to do this, and it is slow as heck.

How can i speed this up? My intuition is that (except for bying a SSD), i could schedule all reads at once, and let the OS find an order which minimizes seek time. However, i'm not sure how to implement this in python.

• maybe spwan 1000 threads, each of which performs a read?
• is there asynchronous numpy.load or equivalent?

Any help is appreciated! Thanks :)

Answer 1

A very simple example of the simple approach from my comment:

from threading import Thread
import timeit
arr=[]
def populate(content):
    for i in content:
        arr.append(i)
content1=[i for i in range(1,10000000)]
content2=[i for i in range(10000000,20000001)]

thread1=Thread(target=populate,kwargs={'content':content1})
thread2=Thread(target=populate,kwargs={'content':content2})

start = timeit.default_timer()
thread1.start()
thread2.start()
thread1.join()
thread2.join()
stop = timeit.default_timer()

print("time taken:",stop-start)

It took 3.546857 seconds with threads and 4.7215586 seconds without threads

So, there is a little speed up. You can add more threads to your problem and the times should improve further.

In your case, you will be loading a fraction of selected 1000 files with np.load and then appending this data to your main array