[英]Iterating over df column list and replacing existing keys with their values from a dictionary efficiently python
I have a dict with items probabilities.我有一个带有项目概率的字典。 I have a df with 5 milion rows that looks like that:
我有一个 df 有 500 万行,看起来像这样:
user_id item_list
U1 [I1,I3,I4]
U2 [I5,I4]
and a dict: {'I1': 0.1, 'I4': 0.4, ..}
和一个 dict:
{'I1': 0.1, 'I4': 0.4, ..}
I am trying to go each row and creat a list with probailities, like that:我正在尝试 go 每一行并创建一个具有概率的列表,如下所示:
user_id item_list prob_list
U1 [I1,I3,I4] [0.1,0.4]
U2 [I5,I4] [0.4]
This is my code:这是我的代码:
keys = list(prob_dict.keys())
df['prob_list'] = df.progress_apply(lambda x: get_probability(prob_dict=prob_dict,
keys=keys, item_list=x['item_list']),axis=1)
def get_probability(prob_dict, keys, item_list):
prob_list = []
for item in item_list:
if item in keys:
prob = prob_dict[item ]
prob_list.append(prob)
if len(prob_list)>=1:
return prob_list
else:
return np.nan
Since I am using tqdm I know how long its going to take (120 hours), which is too much and it's clearly not efficient.由于我使用的是 tqdm,我知道它需要多长时间(120 小时),这太多了,而且显然效率不高。
Any ideas on how I can do it more efficently?关于如何更有效地做到这一点的任何想法?
Use, Series.transform
to transform each item in item_list
to pandas Series and correspondingly map this series using Series.map
to a mapping dictionary d
, then use dropna
to drop the NaN
values:使用
Series.transform
将 item_list 中的每个项目转换为item_list
系列和相应的NaN
这个系列使用Series.map
到映射字典d
,然后使用dropna
值:
d = {'I1': 0.1, 'I4': 0.4}
df['prob_list'] = (
df['item_list'].transform(lambda s: pd.Series(s).map(d).dropna().values)
)
UPDATE (Use multiprocessing
to improve the speed of mapping the item_list
to prob_list
):更新(使用
multiprocessing
来提高映射item_list
到prob_list
的速度):
import multiprocessing as mp
def map_prob(s):
s = s[~s.isna()]
return s.transform(
lambda lst: [d[k] for k in lst if k in d] or np.nan)
def parallel_map(item_list):
splits = np.array_split(item_list, mp.cpu_count())
pool = mp.Pool()
prob_list = pd.concat(pool.map(map_prob, splits))
pool.close()
pool.join()
return prob_list
df['prob_list'] = parallel_map(df['item_list'])
Result:结果:
# print(df)
uer_id item_list prob_list
0 U1 [I1, I3, I4] [0.1, 0.4]
1 U2 [I5, I4] [0.4]
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.