为什么 Dask to_csv 部分保存文件？

Question

Preamble for context:

I have a sample csv file that has more columns than rows (~300 vs 190), I'm trying to learn how it all works before working with the whole 80 million records. I'm working on a google colab notebook.

What I'm trying to do:

Read a CSV file, execute a value_counts() for all the columns and save the results

Here's the code, I left it as is:

import dask.dataframe as dd
import pandas as pd

# Here we're reading the csv
dfd = dd.read_csv(
    'drive/MyDrive/csvs/sample.csv', 
    delimiter=';',

    # Down below we specify the types of the first columns
    dtype = {'ID': object, 'BSID': 'UInt32', 'CAM': 'UInt32',
                  'AGZ': 'UInt32', 'Zen': 'UInt16', 'taw': 'UInt16'
                  },
    blocksize=64000000 # = 64 Mb chunks
)

# Here we convert the rest of the ~300 columns to UInt8

cols=[i for i in dfd.columns if i not in ['ID', 'BSID', 'CAM',
                  'AGZ', 'Zen', 'taw']]
for col in cols:
    dfd[col]=dfd[col].astype('UInt8')

# value_counts
for col in dfd.columns:
  result = dfd[col].value_counts()
  result.to_csv('drive/MyDrive/csvs/Value_counts-' + col + '.csv')

What's going wrong:

When the code is executed, the results are stored as files named 0.part in folders that are named following the Value_counts-' + col + '.csv schema. I expect it to be saved as Value_counts-' + col + '.csv files in csvs folder.

Why is this happening?

Additional question:

Can I run value_counts() for all columns in a better way?

Answer 1

See the doc

single_file:bool, default False

Whether to save everything into a single CSV file. Under the single file mode, each partition is appended at the end of the specified CSV file.

In your case you only have one partition (part.0) for each output - but Dask doesn't know that you don't need parallel writing from multiple chunks, so you need to help it.

Is there a better way? Well, it sounds like you have many more columns than partitions, so you could do dfd.map_partitions(pd.DataFrame.value_counts) and sum the pieces.