数据包发行问题

Question

I have data in key value format. I have created a dask bag and then a dataframe from that bag. But when I am trying do group by on that dataframe its throwing error. But for same data when I directly create a pandas dataframe or dask dataframe it was working fine.

I think I am missing something. Plz help !!!

I have recreated the issue in below code.

import pandas as pd
import dask.dataframe as dd
import dask.bag as db

df = pd.DataFrame({'A': [1, 1, 2, None],  'B': [1, 2, 3, 4]})

df.groupby(df.A).count()  # pandas, working 

ddf = dd.from_pandas(df, 2)
ddf.groupby(ddf.A).count().compute() # dask dataframe, working 

bg = db.from_sequence([{'A': 1,'B':1}, {'A': 1,'B': 2}, {'A': 2,'B':3 }, {'A': None, 'B': 4}])
ddf_2 = bg.to_dataframe()
ddf_2 = ddf_2.fillna(0)
ddf_2.groupby(ddf_2.A).count().compute()  # throws error 

..........
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Note : In actual scenario I have data in avro files. So I cannot skip the dask bag to dataframe part.

Answer 1

The issue is that the dtypes that dask thinks you have aren't the dtypes you actually have. When you Bag.to_dataframe without specifying the output dtypes, dask assumes that the first partition is representative (loading the whole datasset to check is expensive) and infers the dataframe dtypes from that, thus inferring 'A' as an integer column.

In [1]: import dask.bag as db

In [2]: bg = db.from_sequence([{'A': 1,'B':1}, {'A': 1,'B': 2}, {'A': 2,'B':3 }, {'A': None, 'B': 4}])

In [3]: ddf = bg.to_dataframe()

In [4]: ddf.dtypes
Out[4]:
A    int64
B    int64
dtype: object

In actuality though 'A' has a missing value later on, and so can't be an integer column (pandas integer series currently has no missing value representation, you must use floats). To be robust here you should specify the dtypes of the expected dataframe with the meta keyword:

In [5]: ddf = bg.to_dataframe(meta={'A': float, 'B': int})  # specify 'A' has missing values and must be float

In [6]: ddf2 = ddf.fillna(0).astype({'A': int})  # fill missing with 0, and convert A back to int

In [7]: ddf2.groupby(ddf2.A).count().compute()
Out[7]:
   B
A
1  2
2  1
0  1

See the docstring of Bag.to_dataframe for more information.