使用 dask.DataFrame.to_parquet() 写入大文件

Question

I have a.pq file (about 2Gb) in which I want to change a column name using dask.

I have no problems reading the file to dask DataFrame and also I'm able to rename columns. But when it comes to writing the.pq file back to disk with ddf.to_parquet(), the job fails as it seems that dask tries to fit it in memory (and it doesn't fit).

Why does this happen? I expected that dask would do this iteratively. How can I write the target file in chunks?

Below is the code that I'm using.

import dask.dataframe as dd

ddf = dd.read_parquet(
    '/path/to/file/file.pq',
    engine='pyarrow'
)

ddf = ddf.rename(columns={'old_column_name': 'new_column_name'})

# the step which fails
ddf.to_parquet(
    '/path/to/file/edited/',
    engine='pyarrow',
    write_index=False
)

Thanks in advance!

Answer 1

Dask does indeed load your data in chunks, and writes them to the output in chunks. The total memory usage will depend on

• the size of each chunk, known as "row-groups" in parquet, which are not divisible. You need the in-memory size, after decompression and decoding
• the number of chunks you process at once, which may be the number of cores in your CPU if you don't configure otherwise

Note that some intermediate values are also needed during processing, so you generally want each thread of each worker to be able to fit a god deal more than just one chunk's worth of data.