如何在单个列上使用groupby并对Pandas中的多个列执行比较？

Question

I have a dataframe of users, whether or not they have signed up, and the model's prediction for whether or not they have signed up. I want to find per user: the TP (they signed up and the model predicted they did), FP (they didn't sign up but the model predicted they did), FN (they signed up but the model predicted no), and TN (they didn't sign up and the model predicted no). Here 1 means they signed up and 0 means they did not. I want to groupby on users, and then perform comparisons using the other two columns. For example, I might have something like the following:

Users    |    Signed_up    |     Prediction   |
User1         1                  0            
User2         0                  0
User1         1                  1
User3         1                  1
User2         0                  1
User2         0                  0
...

For TP, the resulting table might look something like:

Users    |    TP    |
User1         1
User2         0
User3         1

For TN, the resulting table might look something like:
Users    |    TN    |
User1         0
User2         1
User3         0

and so on for FP and FN.

I am assuming I groupby on the Users column and use a lambda function to compare the Sign_up and Prediction columns, but I am not sure how to actually do this. I would appreciate any help!

Answer 1

Do the comparison before you groupby and then groupby + sum

(df.assign(TP = df.Signed_up & df.Prediction, 
           TN = (df.Signed_up == 0) & (df.Prediction == 0),
           FN = df.Signed_up & (df.Prediction == 0), 
           FP = (df.Signed_up == 0) & df.Prediction)
   .groupby('Users')['TP', 'TN', 'FN', 'FP'].sum())

       TP   TN   FN   FP
Users                   
User1   1  0.0  1.0  0.0
User2   0  2.0  0.0  1.0
User3   1  0.0  0.0  0.0

---

Inspired by @BrianJoseph, with much less typing, you could groupby all 3 columns, determine the size, and unstack everything but the users:

df.groupby([*df]).size().unstack([1,2]).fillna(0)

Signed_up     1         0     
Prediction    0    1    0    1
Users                         
User1       1.0  1.0  0.0  0.0
User2       0.0  0.0  2.0  1.0
User3       0.0  1.0  0.0  0.0

Answer 2

Remember that pandas can groupby using function results. In order to distinguish these 4 classes of results you just need to know the relationship between Signed_up and Prediction . You can classify them like this:

grps = df.groupby(lambda index: (df.loc[index, 'Signed_up'], df.loc[index, 'Prediction']))

This just gives you the groupby object and you can feel free to name groups like:

tp_df = grps.get_group((1,1))

Answer 3

If creating different dfs, for each model prediction which it seems like from your post, you could do this using boolean masking and the & bitwise operator. & means that both conditions must be met to return the value, so:

df = pd.read_csv('./Desktop/models.csv')

TP = df.loc[(df['Signed_up'] == 1) & (df['Prediction'] == 1)]

TN = df.loc[(df['Signed_up'] == 0) & (df['Prediction'] == 0)]

FN = df.loc[(df['Signed_up'] == 1) & (df['Prediction'] == 0)]

FP = df.loc[(df['Signed_up'] == 0) & (df['Prediction'] == 1)]

output:

>>> TP
   Users  Signed_up  Prediction
2  User1          1           1
3  User3          1           1
>>> TN = df.loc[(df['Signed_up'] == 0) & (df['Prediction'] == 0)]
>>> TN
   Users  Signed_up  Prediction
1  User2          0           0
5  User2          0           0
>>> FN = df.loc[(df['Signed_up'] == 1) & (df['Prediction'] == 0)]
>>> FN
   Users  Signed_up  Prediction
0  User1          1           0
>>> FP = df.loc[(df['Signed_up'] == 0) & (df['Prediction'] == 1)]
>>> FP
   Users  Signed_up  Prediction
4  User2          0           1