Panda groupby：计算其他列上满足条件的行？

Question

I would like to do a groupby in pandas obtaining as result a dataframe that has as columns the column used to groupby , the number of elements for each group and among them, the number of elements in each group, the number of elements that does/does not satisfy a condition based on another column value.

For example being the input like this:

type    success
A       True
B       False
A       False
C       True

I would like something like:

type    total    numOfSuccess numOfFailure
A       2        1             1
B       1        0             1
C       1        1             0

In pyspark I did this like

import pyspark.sql.functions as F
df = df.groupBy("type").agg(\
    F.count('*').alias('total'), \
    F.sum(F.when(F.col('success')=="true", 1).otherwise(0)).alias('numOfSuccess'),
    F.sum(F.when(F.col('success')!="true", 1).otherwise(0)).alias('numOfFails'))

while in pandas I can only get the total and numOfSuccess as:

df_new = df.groupby(['type'], as_index=False)['success'].agg({'total':'count', 'numOfSuccess':'sum'})

or only the total as:

df = df.groupby(['type']).size().reset_index(name='NumOfReqs')

but I cannot get the third column numOfFailures and plus if there is an alternative rather than summing the boolean values, it would be better since in can be extended to other cases as well easier in my opinion.

How can I do that?

Answer 1

Use groupby with GroupBy.size for count all data, then for count per catogories need pivoting - with GroupBy.size and unstack , crosstab or pivot_table :

df1 = df.groupby('type').size().reset_index(name='count')
df2 = (df.groupby(['type', 'success']).size().unstack(fill_value=0)
        .rename(columns={True:'numOfSuccess', False:'numOfFails'}))

Alternative for df2 :

df2 = pd.crosstab(df['type'], df['success'])
        .rename(columns={True:'numOfSuccess', False:'numOfFails'}))

Or:

df2 = (df.pivot_table(index='type', columns='success', fill_value=0, aggfunc='size')
        .rename(columns={True:'numOfSuccess', False:'numOfFails'}))

---

df_new = df1.join(df2, on='type')
print (df_new)
  type  count  numOfFails  numOfSuccess
0    A      2           1             1
1    B      1           1             0
2    C      1           0             1

Another solution is use parameter margins in crosstab and remove last row by indexing with iloc :

df = (pd.crosstab(df['type'], df['success'], margins=True)
        .rename(columns={True:'numOfSuccess', False:'numOfFails', 'All':'count'})
        .iloc[:-1]
        .reset_index()
        .rename_axis(None, axis=1))

print (df)
  type  numOfFails  numOfSuccess  count
0    A           1             1      2
1    B           1             0      1
2    C           0             1      1

EDIT: If possible True or False not exist, add reindex for add missing column:

print (df)
  type  success
0    A     True
1    B     True
2    A     True
3    C     True

df1 = df.groupby('type').size().reset_index(name='count')
df2 = (df.groupby(['type', 'success']).size().unstack(fill_value=0)
         .reindex(columns=[True, False], fill_value=0)
         .rename(columns={True:'numOfSuccess', False:'numOfFails'}))


df_new = df1.join(df2, on='type')
print (df_new)
  type  count  numOfSuccess  numOfFails
0    A      2             2           0
1    B      1             1           0
2    C      1             1           0