[英]faster way to run a for loop for a very large dataframe list
I am using two for loops inside each other to calculate a value using combinations of elements in a dataframe list.我在彼此内部使用两个 for 循环,使用 dataframe 列表中的元素组合来计算值。 the list consists of large number of dataframes and using two for loops takes considerable amount of time.
该列表由大量数据帧组成,使用两个 for 循环需要花费大量时间。
Is there a way i can do the operation faster?有什么办法可以更快地完成手术吗?
the functions I refer with dummy names are the ones where I calculate the results.我用虚拟名称引用的函数是我计算结果的函数。
My code looks like this:我的代码如下所示:
conf_list = []
for tr in range(len(trajectories)):
df_1 = trajectories[tr]
if len(df_1) == 0:
continue
for tt in range(len(trajectories)):
df_2 = trajectories[tt]
if len(df_2) == 0:
continue
if df_1.equals(df_2) or df_1['time'].iloc[0] > df_2['time'].iloc[-1] or df_2['time'].iloc[0] > df_1['time'].iloc[-1]:
continue
df_temp = cartesian_product_basic(df_1,df_2)
flg, df_temp = another_function(df_temp)
if flg == 0:
continue
flg_h = some_other_function(df_temp)
if flg_h == 1:
conf_list.append(1)
My input list consist of around 5000 dataframes looking like (having several hundreds of rows)我的输入列表包含大约 5000 个数据框,看起来像(有几百行)
id ![]() |
x ![]() |
y![]() |
z ![]() |
time![]() |
---|---|---|---|---|
1 ![]() |
5 ![]() |
7 ![]() |
2 ![]() |
5 ![]() |
and what i do is I get the cartesian product with combinations of two dataframes and for each couple I calculate another value 'c'.我所做的是通过两个数据帧的组合获得笛卡尔积,并且为每对夫妇计算另一个值“c”。 If this value c meets a condition then I add an element to my c_list so that I can get the final number of couples meeting the requirement.
如果这个值 c 满足条件,那么我将一个元素添加到我的 c_list 中,以便我可以获得满足要求的最终夫妻数量。
For further info;了解更多信息;
a_function(df_1, df_2) is a function getting the cartesian product of two dataframes. a_function(df_1, df_2) 是一个 function 获取两个数据帧的笛卡尔积。
another_function looks like this: another_function 看起来像这样:
def another_function(df_temp):
df_temp['z_dif'] = nwh((df_temp['time_x'] == df_temp['time_y'])
, abs(df_temp['z_x']- df_temp['z_y']) , np.nan)
df_temp = df_temp.dropna()
df_temp['vert_conf'] = nwh((df_temp['z_dif'] >= 1000)
, np.nan , 1)
df_temp = df_temp.dropna()
if len(df_temp) == 0:
flg = 0
else:
flg = 1
return flg, df_temp
and some_other_function looks like this: some_other_function 看起来像这样:
def some_other_function(df_temp):
df_temp['x_dif'] = df_temp['x_x']*df_temp['x_y']
df_temp['y_dif'] = df_temp['y_x']*df_temp['y_y']
df_temp['hor_dif'] = hypot(df_temp['x_dif'], df_temp['y_dif'])
df_temp['conf'] = np.where((df_temp['hor_dif']<=5)
, 1 , np.nan)
if df_temp['conf'].sum()>0:
flg_h = 1
return flg_h
The following are the way to make your code run faster:以下是让你的代码运行得更快的方法:
for-loop
use list comprehension.for-loop
使用列表理解。map
, filter
, sum ect, this would make your code faster.map
、 filter
、 sum 等,这将使您的代码更快。Import datetime
A=datetime.datetime.now() #dont use this
From datetime.datetime import now as timenow
A=timenow()# use this
True
"True
”if-else
to check a Boolean value, avoid using assignment operator.if-else
检查 Boolean 值时,避免使用赋值运算符。# Instead of Below approach
if a==1:
print('a is 1')
else:
print('a is 0')
# Try this approach
if a:
print('a is 1')
else:
print('a is 0')
# This would help as a portion of time is reduce which was used in check the 2 values.
Usefull references:有用的参考:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.