Dask to_csv 生成无法访问的文件

Question

I'm new to Dask. My motivation was to read large CSV files faster by parallelizing the process. After reading a file, I use compute() in order to merge the parts into a single pandas df. Then, when using pandas to_csv , the output CSV file isn't readable:

$ file -I *.csv
my_big_file.csv:               ERROR: cannot read `my_big_file.csv' (Operation canceled)

$ head -n2 my_big_file.csv
head: Error reading my_big_file.csv

Original code looks like the following:

import pandas as pd
import dask.dataframe as daf

filepath='/Users/coolboy/Customer Data/my_original_file.csv'
df = daf.read_csv(filepath, dtype=str, low_memory=False, encoding='utf-8-sig',error_bad_lines=False).compute() 
print('done reading')
df.to_csv('/Users/coolboy/Customer Data/my_big_file.csv',index=False)

Answer 1

The original motivation is to read the data into memory faster. Using dask is a plausible solution, but if the intention is to bring the data into memory, then there are other alternatives available also. For example, modin follows pandas API and could deliver reduction proportional to the number of cores, see docs . The code would roughly look like this:

import modin.pandas as pd
df_in = pd.read_csv(path_in, **options)
... # potentially some additional logic
df_out.to_csv(path_out, **other_options)

If speed/memory efficiency is of primary concern and there is no data transformation happening, then the best alternative is to use shell commands or use Python-based libraries to copy file with pathlib or, if remote data is involved, fsspec .