[英]Python continuous parallel execution
Looking to build a python script that runs an infinite read loop from stdin like for line in sys.stdin:
.希望构建一个 python 脚本,该脚本从 stdin 运行无限读取循环,例如for line in sys.stdin:
。 For each iteration, I would like to get a worker from a pool that executes in the background using line
as input.对于每次迭代,我想从使用line
作为输入在后台执行的池中获取一个 worker。 The process on finishing its execution or timing out prints to stdout.完成其执行或超时的过程打印到标准输出。
I am having a difficult time finding a worker pool module that is able to work continuously.我很难找到能够连续工作的工作池模块。 For example, the multiprocess pool module only supports functions like join
that wait for all workers to finish all tasks.例如, 多进程池模块仅支持join
等功能,等待所有 worker 完成所有任务。 For the above specification, I cannot know all the tasks ahead of time and need to assign work as it comes to processes in the background.对于上述规范,我无法提前知道所有任务,需要在后台分配工作。
This will run forever.这将永远运行。
import sys
from multiprocessing import Pool
pool = Pool()
for line in sys.stdin.readline():
pool.apply_async(function, args=[line])
def function(line):
"""Process the line in a separate process."""
print(line)
Using Pool
and imap
might make it easier, but you have to assume a maximum capacity of workers ( processes=5
):使用Pool
和imap
可能会更容易,但您必须假设工作人员的最大容量( processes=5
):
import multiprocessing
import sys
def worker(line):
return "Worker got %r" % (line)
pool = multiprocessing.Pool(processes=5)
for result in pool.imap(worker, sys.stdin):
print "Result: %r" % (result)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.