python用条件重新编码csv

Question

I am a beginner in python. I need to recode a CSV file:

unique_id,pid,Age
1,1,1
1,2,3
2,1,5
2,2,6
3,1,6
3,2,4
3,3,6
3,4,1
3,5,4
4,1,6
4,2,5

The condition is: for each [unique_id], if there is any [Age]==6, then put a value 1 in the corresponding rows of with a [pid]=1, others should be 0.

the output csv will look like this:

unique_id,pid,Age,recode
1,1,1,0
1,2,3,0
2,1,5,1
2,2,6,0
3,1,6,1
3,2,4,0
3,3,6,0
3,4,1,0
3,5,4,0
4,1,6,1
4,2,5,0

I was using numpy: like follwoing:

import numpy
input_file1 = "data.csv"
input_folder = 'G:/My Drive/'
Her_HH =pd.read_csv(input_folder + input_file1)
Her_HH['recode'] = numpy.select([Her_PP['Age']==6,Her_PP['Age']<6], [1,0], default=Her_HH['recode'])

Her_HH.to_csv('recode_elderly.csv', index=False)

but it does not put value 1 in where [pid] is 1. Any help will be appreciated.

Answer 1

You can use DataFrame.assign for new column with GroupBy.transform for test if at least one match by GroupBy.any , chain mask for test 1 with & for bitwise AND and last cast output to integers

#sorting if necessary
df = df.sort_values('unique_id')

m1 = df.assign(test=df['Age'] == 6).groupby('unique_id')['test'].transform('any')

Another idea for get groups with 6 is filter them with unique_id and Series.isin :

m1 = df['unique_id'].isin(df.loc[df['Age'] == 6, 'unique_id'])

---

m2 = df['pid'] == 1

df['recode'] = (m1 & m2).astype(int)
print (df)
    unique_id  pid  Age  recode
0           1    1    1       0
1           1    2    3       0
2           2    1    5       1
3           2    2    6       0
4           3    1    6       1
5           3    2    4       0
6           3    3    6       0
7           3    4    1       0
8           3    5    4       0
9           4    1    6       1
10          4    2    5       0

EDIT:

For check groups with no match 6 in Age column is possible filter by inverted mask by ~ and if want only all unique rows by unique_id values add DataFrame.drop_duplicates :

print (df[~m1])
   unique_id  pid  Age
0          1    1    1
1          1    2    3

df1 = df[~m1].drop_duplicates('unique_id')
print (df1)
   unique_id  pid  Age
0          1    1    1

Answer 2

This a bit clumsy, since I know numpy a lot better than pandas .

Load your csv sample into a dataframe:

In [205]: df = pd.read_csv('stack59885878.csv')                                                  
In [206]: df                                                                                     
Out[206]: 
    unique_id  pid  Age
0           1    1    1
1           1    2    3
2           2    1    5
3           2    2    6
4           3    1    6
5           3    2    4
6           3    3    6
7           3    4    1
8           3    5    4
9           4    1    6
10          4    2    5

Generate a groupby object based on the unique_id column:

In [207]: gps = df.groupby('unique_id')                                                          

In [209]: gps.groups                                                                             
Out[209]: 
{1: Int64Index([0, 1], dtype='int64'),
 2: Int64Index([2, 3], dtype='int64'),
 3: Int64Index([4, 5, 6, 7, 8], dtype='int64'),
 4: Int64Index([9, 10], dtype='int64')}

I've seen pandas ways for iterating on groups, but here's a list comprehension. The iteration produce a tuple, with the id and a dataframe. We want to test each group dataframe for 'Age' and 'pid' values:

In [211]: recode_values = [(gp['Age']==6).any() & (gp['pid']==1) for x, gp in gps]               
In [212]: recode_values                                                                          
Out[212]: 
[0    False
 1    False
 Name: pid, dtype: bool, 2     True
 3    False
 Name: pid, dtype: bool, 4     True
 5    False
 6    False
 7    False
 8    False
 Name: pid, dtype: bool, 9      True
 10    False
 Name: pid, dtype: bool]

The result is a list of Series, with a True where pid is 1 and there's a 'Age' 6 in the group.

Joining these Series with numpy.hstack produces a boolean array, which we can convert to an integer array:

In [214]: np.hstack(recode_values)                                                               
Out[214]: 
array([False, False,  True, False,  True, False, False, False, False,
        True, False])
In [215]: df['recode']=_.astype(int)            # assign that to a new column
In [216]: df                                                                                     
Out[216]: 
    unique_id  pid  Age  recode
0           1    1    1       0
1           1    2    3       0
2           2    1    5       1
3           2    2    6       0
4           3    1    6       1
5           3    2    4       0
6           3    3    6       0
7           3    4    1       0
8           3    5    4       0
9           4    1    6       1
10          4    2    5       0

Again, I think there's an idiomatic pandas way of joining those series. But for now this works.

===

OK, the groupby object has an apply :

In [223]: def foo(gp): 
     ...:     return (gp['Age']==6).any() & (gp['pid']==1).astype(int) 
     ...:                                                                                        
In [224]: gps.apply(foo)                                                                         
Out[224]: 
unique_id    
1          0     0
           1     0
2          2     1
           3     0
3          4     1
           5     0
           6     0
           7     0
           8     0
4          9     1
           10    0
Name: pid, dtype: int64

And remove the multi-indexing with:

In [242]: gps.apply(foo).reset_index(0, True)                                                    
Out[242]: 
0     0
1     0
2     1
3     0
4     1
5     0
6     0
7     0
8     0
9     1
10    0
Name: pid, dtype: int64
In [243]: df['recode']=_     # and assign to recode

Lots of experimenting and learning here.