How to read a compressed (gz) CSV file into a dask Dataframe?

Question

Is there a way to read a .csv file that is compressed via gz into a dask dataframe?

I've tried it directly with

import dask.dataframe as dd
df = dd.read_csv("Data.gz" )

but get an unicode error (probably because it is interpreting the compressed bytes) There is a "compression" parameter but compression = "gz" won't work and I can't find any documentation so far.

With pandas I can read the file directly without a problem other than the result blowing up my memory ;-) but if I restrict the number of lines it works fine.

import pandas.Dataframe as pd
df = pd.read_csv("Data.gz", ncols=100)

Answer 1

Panda's current documentation says:

compression : {'infer', 'gzip', 'bz2', 'zip', 'xz', None}, default 'infer'

Since 'infer' is the default, that would explain why it is working with pandas.

Dask's documentation on the compression argument:

String like 'gzip' or 'xz'. Must support efficient random access. Filenames with extensions corresponding to known compression algorithms (gz, bz2) will be compressed accordingly automatically

That would suggest that it should also infer the compression for at least gz . That it doesn't (and it still does not in 0.15.3) may be a bug. However, it is working using compression='gzip'.

ie:

import dask.dataframe as dd
df = dd.read_csv("Data.gz", compression='gzip')

Answer 2

It's actually a long-standing limitation of dask . Load the files with dask.delayed instead :

import pandas as pd
import dask.dataframe as dd
from dask.delayed import delayed

filenames = ...
dfs = [delayed(pd.read_csv)(fn) for fn in filenames]

df = dd.from_delayed(dfs) # df is a dask dataframe

Answer 3

Without the file it's difficult to say. what if you set the encoding like # -*- coding: latin-1 -*- ? or since read_csv is based off of Pandas, you may even dd.read_csv('Data.gz', encoding='utf-8') . Here's the list of Python encodings: https://docs.python.org/3/library/codecs.html#standard-encodings