遍历两个数据帧，并用 pandas 中的第二个数据帧的一列更新第一个数据帧的一列

Question

I am converting a piece of code written in R to python. The following code is in R. df1 and df2 are the dataframes. id , case , feature , feature_value are column names. The code in R is

for(i in 1:dim(df1)[1]){
 temp = subset(df2,df2$id == df1$case[i],select = df1$feature[i])
 df1$feature_value[i] = temp[,df1$feature[i]]
 }

My code in python is as follows.

for i in range(0,len(df1)):
   temp=np.where(df1['case'].iloc[i]==df2['id']),df1['feature'].iloc[i]                                  
   df1['feature_value'].iloc[i]=temp[:,df1['feature'].iloc[i]]

but it gives

TypeError: tuple indices must be integers or slices, not tuple

How to rectify this error? Appreciate any help.

Answer 1

Unfortunately, R and Pandas handle dataframes pretty differently. If you'll be using Pandas a lot, it would probably be worth going through a tutorial on it.

I'm not too familiar with R so this is what I think you want to do: Find rows in df1 where the 'case' matches an 'id' in df2. If such a row is found, add the "feature" in df1 to a new df1 column called "feature_value." If so, you can do this with the following:

#create a sample df1 and df2
>>> df1 = pd.DataFrame({'case': [1, 2, 3], 'feature': [3, 4, 5]})
>>> df1
   case  feature
0     1        3
1     2        4
2     3        5

>>> df2 = pd.DataFrame({'id': [1, 3, 7], 'age': [45, 63, 39]})
>>> df2
   id  age
0   1   45
1   3   63
2   7   39

#create a list with all the "id" values of df2
>>> df2_list = df2['id'].to_list()
>>> df2_list
[1, 3, 7]


#lambda allows small functions; in this case, the value of df1['feature_value']
#for each row is assigned df1['feature'] if df1['case'] is in df2_list, 
#and otherwise it is assigned np.nan.

>>> df1['feature_value'] = df1.apply(lambda x: x['feature'] if x['case'] in df2_list else np.nan, axis=1)
>>> df1
   case  feature  feature_value
0     1        3            3.0
1     2        4            NaN
2     3        5            5.0

Instead of lamda, a full function can be created, which may be easier to understand:

def get_feature_values(df, id_list):
    
    if df['case'] in id_list:
        feature_value = df['feature']
    else:
        feature_value = np.nan
        
    return feature_value

df1['feature_value'] = df1.apply(get_feature_values, id_list=df2_list, axis=1)

Another way of going about this would involve merging df1 and df2 to find rows where the "case" value in df1 matches an "id" value in df2 ( https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html )

===================

To address the follow-up question in the comments: You can do this by merging the databases and then creating a function.

#create example dataframes
>>> df1 = pd.DataFrame({'case': [1, 2, 3], 'feature': [3, 4, 5], 'names': ['a', 'b', 'c']})
>>> df2 = pd.DataFrame({'id': [1, 3, 7], 'age': [45, 63, 39], 'a': [30, 31, 32], 'b': [40, 41, 42], 'c': [50, 51, 52]})

#merge the dataframes
>>> df1 = df1.merge(df2, how='left', left_on='case', right_on='id')
>>> df1
   case  feature names   id   age     a     b     c
0     1        3     a  1.0  45.0  30.0  40.0  50.0
1     2        4     b  NaN   NaN   NaN   NaN   NaN
2     3        5     c  3.0  63.0  31.0  41.0  51.0

Then you can create the following function:

def get_feature_values_2(df):
    
    if pd.notnull(df['id']):
        feature_value = df['feature']
        column_of_interest = df['names']
        feature_extended_value = df[column_of_interest]
    else:
        feature_value = np.nan
        feature_extended_value = np.nan
        
    return feature_value, feature_extended_value


# "result_type='expand'" allows multiple values to be returned from the function
df1[['feature_value', 'feature_extended_value']] = df1.apply(get_feature_values_2, result_type='expand', axis=1)


#This results in the following dataframe:
   case  feature names   id   age     a     b     c  feature_value  \
0     1        3     a  1.0  45.0  30.0  40.0  50.0            3.0   
1     2        4     b  NaN   NaN   NaN   NaN   NaN            NaN   
2     3        5     c  3.0  63.0  31.0  41.0  51.0            5.0   

   feature_extended_value  
0                    30.0  
1                     NaN  
2                    51.0


#To keep only a subset of the columns:
#First create a copy-pasteable list of the column names
list(df1.columns)
['case', 'feature', 'names', 'id', 'age', 'a', 'b', 'c', 'feature_value', 'feature_extended_value']

#Choose the subset of columns you would like to keep
df1 = df1[['case', 'feature', 'names', 'feature_value', 'feature_extended_value']]

df1
   case  feature names  feature_value  feature_extended_value
0     1        3     a            3.0                    30.0
1     2        4     b            NaN                     NaN
2     3        5     c            5.0                    51.0