重新分区镶木地板文件 dask

Question

I want to understand a few things about partioning a parquet on Dask.

When I do it in a.csv file, the chunksize works as intended, doing 30 partitions based on 50 mb chunks.

When I try to do it the same logic through the read_parquet, none partition is created, and when I force this with repartition(partition_size='50mb'), it create 109 partitions.

Can someone explain to me why parquet doesn't seems to work at the same way like.csv when doing chunksizes?

Answer 1

In CSV, the fundamental, non-splittable chunk of data is one row, usually the bytes between one \n character and the subsequent one. This bytes chunk size is typically small. When you load data with dask, it reads from a given offset to the next \n to be able to read an exact number of rows. You would find, if you made the chunk size too small, that some partitions would contain no data.

Parquet is not structured like this. Its fundamental non-splittable chunk is the "row-group", and there is often just one row group per data file. This is done for efficiency: encoding and compressing a whole row group's worth of data in one block will give maximum read throughput. Furthermore, because of the encoding and compression, it's much harder for dask to guess how big a piece of a dataset will be as an in-memory pandas dataframe, but it can be many times bigger.

A row group could easily be >>100MB in size. In fact, this is generally recommended, as smaller pieces will have a higher fraction of their processing time in overhead and latency.

To summarize

• dask will not split a parquet dataset beyond the partitioning within the data files
• that partition size might be many times larger in memory than on disk, so repartitioning after load may result in many partitions
• these are tradeoffs required to make parquet as fast and space-efficient as it is