Pandas 在 groupby 中返回空组

Question

I have a Pandas DataFrame with 3 columns, target , pred , and conf_bin . If I run a groupby(by='conf_bin').apply(...) my apply function gets called with empty DataFrame s for values that do not appear in the conf_bin column. How is this possible?

---

Details

The DataFrame looks something like this:

        target  pred conf_bin
0            5     6     0.50
1            4     4     0.60
2            4     4     0.50
3            4     3     0.50
4            4     5     0.50
5            5     5     0.55
6            5     5     0.55
7            5     5     0.55

Obviously conf_bin is a numeric bin with values in the range np.arange(0, 1, 0.05) . However, not all values are present in the data:

In [224]: grp = tp.groupby(by='conf_bin')

In [225]: grp.groups.keys()
Out[225]: dict_keys([0.5, 0.60000000000000009, 0.35000000000000003, 0.75, 0.85000000000000009, 0.65000000000000002, 0.55000000000000004, 0.80000000000000004, 0.20000000000000001, 0.45000000000000001, 0.40000000000000002, 0.30000000000000004, 0.70000000000000007, 0.25])

So, for example, the values 0 and 0.05 do not appear. However, when I run an apply on the group my function does get called for these values:

In [226]: grp.apply(lambda x: x.shape)
Out[226]:
conf_bin
0.00        (0, 3)
0.05        (0, 3)
0.10        (0, 3)
0.15        (0, 3)
0.20       (22, 3)
0.25       (75, 3)
0.30       (95, 3)
0.35      (870, 3)
0.40     (8505, 3)
0.45    (40068, 3)
0.50    (51238, 3)
0.55    (54305, 3)
0.60    (47191, 3)
0.65    (38977, 3)
0.70    (34444, 3)
0.75    (20435, 3)
0.80     (3352, 3)
0.85        (4, 3)
0.90        (0, 3)
dtype: object

Questions:

1. How can Pandas even know that the values 0.0 and 0.5 "make sense" since they don't appear in my DataFrame ?
2. Why is it calling my apply function with empty DataFrame objects for values that do no appear in grp.groups ?

Answer 1

I too was having this problem, which popped up when trying to create subplots for every category in my dataframe.

I came up with the following workaround (based on this SO post ), by pulling out the non-empty groups into a list.

groups = df.groupby('conf_bin')
group_list = [(index, group) for index, group in groups if len(group) > 0]

It does break the implicit contract that "you wrangle your data in pandas", and probably mismanages memory, but it works.

---

Now you can iterate through your groupby list with the same interface as with a groupby object, eg

fig, axes = plt.subplots(nrows=len(group_list), ncols=1)
for (index, group), ax in zip(group_list, axes.flatten()):
    group['target'].plot(ax=ax, title=index)

Answer 2

The main advantages of using categorical dtype are:

• Memory efficiency. The data is stored as integer codes, which are smaller in size than strings, the category type requires less memory to store the same amount of data compared to object type or int type data.
• Faster processing. Categorical data operations such as group by are generally faster than equivalent operations on object or int type data because they can be performed on the integer codes, which are more efficient to work with than strings.

The cons are:

• group by output: the output of the groupby is very messy. a lof of Nan are generated depending on your categories values.
• the same problem applies to the filtering.
• concatenation issue with category type: the category type is linked to a dictionary of values so when you concatenate or merge you will have trouble and the loss of the category dtype.

You could have more in-depth information from this article: https://medium.com/gitconnected/pandas-category-type-pros-and-cons-1bcac1bdea71