在大型 CSV 文件中查找 #ROWS

Question

The aim is to find the total number of rows in a large CSV file. I m using Python Dask to find it for now, but as the file size is around 45G it takes quite some time. Unix cat with wc -l seems to perform better.

So the question is - Are there any tweaks for dask / pandas read_csv to make it find the total numbers of rows faster?

Answer 1

Dask dataframe will spend 90% of its time parsing your text into various numerical types like int, float, etc.. You don't need any of this, so it's best not to make anything like a dataframe.

You could use dask.bag, which would be faster/simpler

dask.bag.read_text("...").count().compute()

But in truth wc -l is going to be about as fast as anything else. You should be entirely bound by your disk speeds here, and not by compute power. Dask helps you to leverage multiple cores on your CPU, but those aren't the bottleneck in this case, so Dask isn't the right tool, wc is.

Answer 2

You can try subprocess in python code:

fileName = "file.csv"
cmd = 'wc -l {0}'.format(fileName)
output = subprocess.call(cmd)