Pandas: change cell values based on condition

Question

I have the following Pandas dataframe.

import pandas as pd

data = {'id_a': [1, 1, 1, 2, 2, 2, 3, 4], 'name_a': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'd'], 
        'id_b': [5, 6, 7, 8, 9, 10, 11, 11], 'name_b': ['e', 'f', 'g', 'h', 'i', 'j', 'k', 'k'], 
        'similar': [1, 1, 1, 1, 1, 0, 1, 1], 'metric': [.5, 1, .8, .7, .2, .9, .8, .9]}
df = pd.DataFrame(data)
print(df)

⠀

      id_a   name_a   id_b   name_b   similar   metric  
 --- ------ -------- ------ -------- --------- -------- 
  0    1       a       5       e         1       0.5    
  1    1       a       6       f         1       1.0    
  2    1       a       7       g         1       0.8    
  3    2       b       8       h         1       0.7    
  4    2       b       9       i         1       0.2    
  5    2       b       10      j         0       0.9    
  6    3       c       11      k         1       0.8    
  7    4       d       11      k         1       0.9   

In this table, the IDs of group A are linked to the IDs of group B (based on column similar ).

But I need a unique ID of each group to correspond to only one ID of another group.

And among the rows with the same ID of each group, I need to select the row in which the column metric is maximum.

For example, I have three rows with id_a == 2. Among these three rows, only two have a column similar value equal to 1. Among these two rows, one row has a column metric value of 0.7, and the second one has 0.2.

I leave the value of column similar = 1, only for the row with a column metric of 0.7 (because it is maximum), and for the second row I put the value of column similar = 0.

That is, I need the following dataframe:

output_data = {'id_a': [1, 1, 1, 2, 2, 2, 3, 4], 'name_a': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'd'], 
               'id_b': [5, 6, 7, 8, 9, 10, 11, 11], 'name_b': ['e', 'f', 'g', 'h', 'i', 'j', 'k', 'k'], 
               'similar': [0, 1, 0, 1, 0, 0, 0, 1], 'metric': [.5, 1, .8, .7, .2, .9, .8, .9]}
output_df = pd.DataFrame(output_data)
print(output_df)

⠀

      id_a   name_a   id_b   name_b   similar   metric  
 --- ------ -------- ------ -------- --------- -------- 
  0    1       a       5       e         0       0.5    
  1    1       a       6       f         1       1.0    
  2    1       a       7       g         0       0.8    
  3    2       b       8       h         1       0.7    
  4    2       b       9       i         0       0.2    
  5    2       b       10      j         0       0.9    
  6    3       c       11      k         0       0.8    
  7    4       d       11      k         1       0.9    

Question: How to implement this using Python (because my research did not give any results)?

Answer 1

I'm not sure about how you're handling the case of id_a == 3 for instnace but I think this is what you want. Just take the max index from each group (grouped by id_a ) and then, after resetting the similar column, reset those maximal indexes to 1.

max_vals = df.groupby('id_a').apply(lambda grp: grp.loc[grp['similar'] == 1, 'metric'].idxmax())
df['similar'] = 0
df.loc[max_vals, 'similar'] = 1

>>> df

    id_a    name_a  id_b    name_b  similar metric
0   1       a       5       e       0       0.5
1   1       a       6       f       1       1.0
2   1       a       7       g       0       0.8
3   2       b       8       h       1       0.7
4   2       b       9       i       0       0.2
5   2       b       10      j       0       0.9
6   3       c       11      k       1       0.8
7   4       d       11      k       1       0.9

EDIT : See the comments as to why the output doesn't match exactly for row #6.

Answer 2

IIUC, you could do:

# find the indices of the maximum by id_a
keep_a = df[df.similar.eq(1)].groupby('id_a').filter(lambda x: len(x) > 1).groupby('id_a').metric.idxmax()

# find the indices of the maximum by id_b
keep_b = df[df.similar.eq(1)].groupby('id_b').filter(lambda x: len(x) > 1).groupby('id_b').metric.idxmax()

# create mask False if is in set of maximum
mask = ~df.index.isin(set(keep_a) | set(keep_b))

# set values using mask
df.loc[mask, 'similar'] = 0

print(df)

Output

   id_a name_a  id_b name_b  similar  metric
0     1      a     5      e        0     0.5
1     1      a     6      f        1     1.0
2     1      a     7      g        0     0.8
3     2      b     8      h        1     0.7
4     2      b     9      i        0     0.2
5     2      b    10      j        0     0.9
6     3      c    11      k        0     0.8
7     4      d    11      k        1     0.9

Answer 3

Here is a clear symmetrical, orderly and fast way to do this task.

• Series.mask to transform the value of metric into NaN where similar == 0 so that it can never be the maximum and therefore have a 1 in the result.

• Series.shift + Series.cumsum + Series.all to be able to group when there are either consecutive values in id_a or in id_b . Keep in mind that this would be that simple for N ids.

• create a series with the maximums by groups using groupby.transform and compare it with the Metric Series to obtain a Boolean series that you can convert with Series.astype to 1 or 0

---

df2=df.copy()
#discarding similar == 0 as a maximum candidate in the groups
df2['metric']=df2['metric'].mask(df2['similar'].eq(0))

#creating groups depend on id_a and id_b
ids=df2[['id_a','id_b']]
groups=ids.ne(ids.shift()).all(axis=1).cumsum()

#checking the maximum per group and converting to integer
df['similar']=df['metric'].eq(df2.groupby(groups).metric.transform('max')).astype(int)
print(df)

---

Output

   id_a name_a  id_b name_b  similar  metric
0     1      a     5      e        0     0.5
1     1      a     6      f        1     1.0
2     1      a     7      g        0     0.8
3     2      b     8      h        1     0.7
4     2      b     9      i        0     0.2
5     2      b    10      j        0     0.9
6     3      c    11      k        0     0.8
7     4      d    11      k        1     0.9

---

Detail of groups

print(groups)
0    1
1    1
2    1
3    2
4    2
5    2
6    3
7    3
dtype: int64

Answer 4

Use groupby idxmax , isin and on 2 groupby's within the listcomp and passing to np.array . Finally, call all and astype on np.array

df1 = df[df.similar.eq(1)]
df['similar'] = np.array([df.index.isin(df1.groupby(col).metric.idxmax()) 
                            for col in ['id_a', 'id_b']]).all(0).astype(int)


Out[132]:
   id_a name_a  id_b name_b  similar  metric
0     1      a     5      e        0     0.5
1     1      a     6      f        1     1.0
2     1      a     7      g        0     0.8
3     2      b     8      h        1     0.7
4     2      b     9      i        0     0.2
5     2      b    10      j        0     0.9
6     3      c    11      k        0     0.8
7     4      d    11      k        1     0.9

Answer 5

A solution which uses vectorized methods only.

• m1 : vector with max values per group and similar == 1
• m2 : rows where similar == 1
• m3 : rows which have max value & similar == 1

m1 = df.query('similar == 1').groupby('id_a')['metric'].transform('max')
m2 = df['similar'].eq(1)
m3 = df.loc[m2, 'metric'].eq(m1)

df.loc[m3[~m3].index, 'similar'] = 0

   id_a name_a  id_b name_b  similar  metric
0     1      a     5      e        0    0.50
1     1      a     6      f        1    1.00
2     1      a     7      g        0    0.80
3     2      b     8      h        1    0.70
4     2      b     9      i        0    0.20
5     2      b    10      j        0    0.90
6     3      c    11      k        1    0.80
7     4      d    11      k        1    0.90