[英]Python: Parallel Processing in Joblib Makes the Code Run Even Slower
I want to integrate a parallel processing to make my for loops run faster. 我想集成一个并行处理,以使我的for循环运行得更快。
However, I noticed that it has just made my code run slower. 但是,我注意到这使我的代码运行速度变慢了。 See below example where I am using joblib
with a simple function on a list of random integers. 请参见下面的示例,其中我在随机整数列表上使用具有简单函数的joblib
。 Notice that without the parallel processing it runs faster than with . 请注意, 没有并行处理,它的运行速度要比使用并行处理的速度快。
Any insight as to what is happening? 对正在发生的事情有什么见解?
def f(x):
return x**x
if __name__ == '__main__':
s = [random.randint(0, 100) for _ in range(0, 10000)]
# without parallel processing
t0 = time.time()
out1 = [f(x) for x in s]
t1 = time.time()
print("without parallel processing: ", t1 - t0)
# with parallel processing
t0 = time.time()
out2 = Parallel(n_jobs=8, batch_size=len(s), backend="threading")(delayed(f)(x) for x in s)
t1 = time.time()
print("with parallel processing: ", t1 - t0)
I am getting the following output: 我得到以下输出:
without parallel processing: 0.0070569515228271484
with parallel processing: 0.10714387893676758
The parameter batch_size=len(s)
effectively says give each process a batch of s jobs. 参数batch_size=len(s)
有效地表示给每个进程一批s作业。 This means you create 8 threads but then give all workload to 1 thread. 这意味着您创建8个线程,然后将所有工作负载分配给1个线程。
Also you might want to increase the workload to have a measurable advantage. 另外,您可能希望增加工作量以具有可衡量的优势。 I prefer to use time.sleep delays: 我更喜欢使用time.sleep延迟:
def f(x):
time.sleep(0.001)
return x**x
out2 = Parallel(n_jobs=8,
#batch_size=len(s),
backend="threading")(delayed(f)(x) for x in s)
without parallel processing: 11.562264442443848 没有并行处理:11.562264442443848
with parallel processing: 1.412865400314331 并行处理:1.412865400314331
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.