![](/img/trans.png)
[英]python multiprocessing.Pool kill *specific* long running or hung process
[英]Python multiprocessing: kill process if it is taking too long to return
下面是一个简单的例子,它冻结了,因为子进程退出时没有返回任何东西,而父进程一直在等待。 如果花费太长时间,有没有办法让进程超时并让其余进程继续? 我是 python 中多处理的初学者,我发现文档不是很有启发性。
import multiprocessing as mp
import time
def foo(x):
if x == 3:
sys.exit()
#some heavy computation here
return result
if __name__ == '__main__':
pool = mp.Pool(mp.cpu_count)
results = pool.map(foo, [1, 2, 3])
我遇到了同样的问题,这就是我解决它的方法。 也许有更好的解决方案,但是,它也解决了未提及的问题。 例如,如果进程占用了许多资源,则可能会发生正常终止需要一段时间才能通过进程 - 因此我使用强制终止( kill -9
)。 这部分可能仅适用于 Linux,因此如果您使用其他操作系统,则可能需要调整终止。
它是我自己代码的一部分,因此可能无法复制粘贴。
from multiprocessing import Process, Queue
import os
import time
timeout_s = 5000 # seconds after which you want to kill the process
queue = Queue() # results can be written in here, if you have return objects
p = Process(target=INTENSIVE_FUNCTION, args=(ARGS_TO_INTENSIVE_FUNCTION, queue))
p.start()
start_time = time.time()
check_interval_s = 5 # regularly check what the process is doing
kill_process = False
finished_work = False
while not kill_process and not finished_work:
time.sleep(check_interval_s)
now = time.time()
runtime = now - start_time
if not p.is_alive():
print("finished work")
finished_work = True
if runtime > timeout_s and not finished_work:
print("prepare killing process")
kill_process = True
if kill_process:
while p.is_alive():
# forcefully kill the process, because often (during heavvy computations) a graceful termination
# can be ignored by a process.
print(f"send SIGKILL signal to process because exceeding {timeout_s} seconds.")
os.system(f"kill -9 {p.pid}")
if p.is_alive():
time.sleep(check_interval_s)
else:
try:
p.join(60) # wait 60 seconds to join the process
RETURN_VALS = queue.get(timeout=60)
except Exception:
# This can happen if a process was killed for other reasons (such as out of memory)
print("Joining the process and receiving results failed, results are set as invalid.")
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.