使用python和numpy加载文件的最快方法是什么？

Question

I want to train a model and I have a big dataset for training. Its size is more than 20gb. But when I try to read it, it took so long time. I mean to load it on memory.

with open(file_path, newline='', encoding='utf-8') as f:
    reader = csv.reader(f)
    for i,row in enumerate(islice(reader,0,1)):
        train_data = np.array(makefloat(row))[None,:]
    for i,row in enumerate(reader):
        train_data = np.vstack((train_data,np.array(makefloat(row))[None,:]))

It has 43 floats for each line.

It took so long time, I tested it for just 100,000 lines and it took 20 mins.

I think I'm doing wrong. How can I make it faster?

Answer 1

Its' not good to read the entire file. You can use something like Dask which will read your file in chunks and will be faster. Dask