I am currently trying to replace information of a dataframe using another dataframe and a series for my simulation analysis.
Toy example is as follows
A is a user info dataframe, B is a service info dataframe, and C is series information about whether the user changed service.
TableA (user's current service info):
cost location
John 100 Tokyo
Tom 50 Seoul
Andy 50 Seoul
Mark 80 Seoul
TableB (service info):
cost location
premium_T 100 Tokyo
basic_T 60 Tokyo
premium_S 80 Seoul
basic_S 50 Seoul
Table C (service change info):
change
John no
Tom no
Andy premium_S
Mark basic_S
using above data, I'd like to change information in Table A, using data in Table B and C. In other words, I desire:
TableA' (modified user's service info):
cost location
John 100 Tokyo
Tom 50 Seoul
Andy 80 Seoul
Mark 50 Seoul
The code I used is:
TableA = pd.DataFrame(index = ['John', 'Tom', 'Andy', 'Mark'],
data = {'cost': [100,50,50,80],
'location': ['Tokyo', 'Seoul', 'Seoul', 'Seoul']})
TableB = pd.DataFrame(index = ['premium_T', 'basic_T', 'premium_S', 'basic_S'],
data = {'cost': [100, 60, 80, 50],
'location': ['Tokyo','Tokyo','Seoul','Seoul']})
TableC = pd.Series( ['no', 'no', 'premium_S', 'basic_S'], index = ['John', 'Tom', 'Andy', 'Mark'])
customer_list = TableA.index.tolist()
for k in customer_list:
if TableC.loc[k] != 'no':
TableA.loc[k] = TableB.loc[TableC.loc[k]]
The code works, and provides the results that I desire.
However, I have to do such work for a very big dataset repeatedly, and I need faster method to do such replacements.
Any ideas? I think repeated use of .loc is the problem, but I have not found probable solution yet. I have looked at pd.update() or pd.replace(), but it does not seem to be what I am looking for.
Thank you in advance
If we convert everything to dataframes with named columns, we can use merges to pull in the correct information:
TableA = TableA.reset_index().rename({'index': 'person'}, axis='columns')
TableB = TableB.reset_index().rename({'index': 'cost_plan'}, axis='columns')
TableC = TableC.to_frame(name='cost_plan').reset_index().rename({'index': 'person'}, axis='columns')
new_costs = TableA.merge(TableC, how='left').merge(TableB, how='left',
on=['location', 'cost_plan'],
suffixes=['_old', '_new'])
new_costs['cost_new'].fillna(new_costs['cost_old'], inplace=True)
new_costs
then looks like:
person cost_old location cost_plan cost_new
0 John 100 Tokyo no 100.0
1 Tom 50 Seoul no 50.0
2 Andy 50 Seoul premium_S 80.0
3 Mark 80 Seoul basic_S 50.0
First calculate in-scope customers from TableC
using reindex
and Boolean indexing:
idx = TableC.reindex(TableA.index & TableC.index)
idx = idx[idx != 'no']
Then update TableA
via loc
:
TableA.loc[np.in1d(TableA.index, idx.index)] = TableB.reindex(idx.values).values
Result:
cost location
John 100.0 Tokyo
Tom 50.0 Seoul
Andy 80.0 Seoul
Mark 50.0 Seoul
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.