[英]Python multiprocessing and progress bar function occasionally hangs
I have this general function that I use to run many different operations that works to launch multiple processes and then keep track of the progress using the progressbar2 modules. 我具有用于运行许多不同操作的通用功能,这些操作可启动多个进程,然后使用progressbar2模块跟踪进度。 Occasionally the progress bar results in a hang up and the function gets stuck. 有时进度条会导致挂断,并且功能会卡住。 I can't seem to figure out why, if I remove the progress bar update, then the process finishes and moves on, so the problem seems to be in the while loop. 我似乎无法弄清楚为什么,如果删除进度条更新,则过程完成并继续进行,因此问题似乎出在while循环中。 Is there a better way to construct the while loop? 有没有更好的方法来构造while循环?
def runMultiProgress(function, inputList, cpus):
from progressbar import ProgressBar, Percentage, ETA
from time import sleep
#setup pool
p = multiprocessing.Pool(cpus)
#setup progress bar
widgets = [' Progress: ', Percentage(),' || ', ETA()]
pbar = ProgressBar(widgets=widgets, term_width=30, maxval=len(inputList)).start()
#setup results and split over cpus
results = []
r = [p.apply_async(function, (x,), callback=results.append) for x in inputList]
#refresh pbar every 5 seconds
while len(results) != len(inputList):
pbar.update(len(results))
sleep(5)
pbar.finish()
p.close()
p.join()
UPDATE: citing my sources, this multiprocessing + progress bar is from @julien-tourille answer here: Show the progress of a Python multiprocessing pool map call? 更新:援引我的资料,这个多处理+进度栏来自@ julien-tourille答案在这里: 显示Python多处理池映射调用的进度?
I ended up modifying the function to use just a simple progressbar and modified how the multiprocessing was being called. 我最终修改了该函数以仅使用一个简单的进度栏,并修改了如何调用多处理。 This seems to be simpler and works for me. 这似乎更简单,并且对我有用。
def runMultiProgress(function, inputList, cpus):
#setup pool
p = multiprocessing.Pool(cpus)
#setup results and split over cpus
tasks = len(inputList)
results = []
for i in inputList:
results.append(p.apply_async(function, [i]))
#refresh progress every sec
while True:
incomplete_count = sum(1 for x in results if not x.ready())
if incomplete_count == 0:
break
sys.stdout.write("Progress: %.2f%% \r" % (float(tasks - incomplete_count) / tasks * 100))
sys.stdout.flush()
time.sleep(1)
p.close()
p.join()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.