如何快速将大型CSV文件读入Python？

Question

I'm trying to load a ~67 gb dataframe (6,000,000 features by 2300 rows) into dask for machine learning. I'm using a 96 core machine on AWS that I wish to utilize for the actual machine learning bit. However, Dask loads CSVs in a single thread. It has already taken a full 24 hours and it hasn't loaded.

#I tried to display a progress bar, but it is not implemented on dask's load_csv
from dask.diagnostics import ProgressBar
pbar = ProgressBar()
pbar.register()

df = dd.read_csv('../Larger_than_the_average_CSV.csv')

Is there a faster way to load this into Dask and make it persistent? Should I switch to a different technology (Spark on Scala or PySpark?)

Dask is probably still loading it as I can see a steady 100% CPU utilization in top .

Answer 1

The code you show in the question probably takes no time at all, because you are not actually loading anything, just setting up the job prescription. How long this takes will depend on the chunksize you specify.

There are two main bottlenecks to consider for actual loading:

• getting the data from disc into memory, raw data transfer over a single disc interface,
• parsing that data into in-memory stuff

There is not much you can do about the former if you are on a local disc, and you would expect it to be a small fraction.

The latter may suffer from the GIL, even though dask will execute in multiple threads by default (which is why it may appear only one thread is being used). You would do well to read the dask documentation about the different schedulers, and should try using the Distributed scheduler, even though you are on a single machine, with a mix of threads and processes.

Finally, you probably don't want to "load" the data at all, but process it. Yes, you can persist into memory with Dask if you wish ( dask.persist , funnily), but please do not use many workers to load the data just so you then make it into a Pandas dataframe in your client process memory.