[英]Why does multiprocessing.Process.join() hang?
I am using multiprocessing in this manner: 我正在以这种方式使用多处理:
import multiprocessing as mp
def worker(thread_id, tasks, results):
tmp_dir = 'temp_for_{}'.format(thread_id)
os.makedirs(tmp_dir)
try:
while not tasks.empty():
data = tasks.get()
response = process_pdf(data, tmp_dir)
results.put(response)
except (KeyboardInterrupt, SystemExit):
log.info('Interrupt signal received in thread %s.', thread_id)
except Queue.Empty:
pass
except Exception:
log.error("Unexpected error in %s", thread_id, exc_info=True)
finally:
shutil.rmtree(tmp_dir)
log.info("Thread %s exit", thread_id)
if __name__ == "__main__":
tasks, results = mp.Queue(), mp.Queue()
for record in cursor.select(query):
tasks.put(record)
manager = mp.Manager()
workers = [mp.Process(target=worker, args=(i, tasks, results)) for i in xrange(8)]
for worker in workers:
worker.start()
try:
for worker in workers:
worker.join()
except (KeyboardInterrupt, SystemExit):
log.info('Interrupt signal received in main. Cleaning up main')
finally:
log.info('Got %s results. Saving', results.qsize())
while not results.empty():
cursor.update_one('documents', 'id', results.get())
cursor.close()
Here's the output when I run this code: 这是我运行此代码时的输出:
14:34:04 15/10 INFO: Thread 6 exit
14:34:04 15/10 INFO: Thread 7 exit
14:34:21 15/10 INFO: Thread 3 exit
14:34:24 15/10 INFO: Thread 2 exit
14:34:24 15/10 INFO: Thread 1 exit
14:34:29 15/10 INFO: Thread 5 exit
14:34:36 15/10 INFO: Thread 0 exit
14:35:37 15/10 INFO: Thread 4 exit
Then I enter ^C after waiting for a while with no progress, and get this output: 然后我在等待一段时间没有进展后输入^ C,并获得此输出:
^C14:37:16 15/10 INFO: Interrupt signal received in main. Cleaning up main
14:37:16 15/10 INFO: Got 16 results. Saving
And I get this traceback for all threads: 我得到了所有线程的回溯:
Process Process-9:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 261, in _bootstrap
util._exit_function()
File "/usr/lib64/python2.7/multiprocessing/util.py", line 328, in _exit_function
util._exit_function()
File "/usr/lib64/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
finalizer()
File "/usr/lib64/python2.7/multiprocessing/util.py", line 207, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib64/python2.7/multiprocessing/queues.py", line 218, in _finalize_join
thread.join()
File "/usr/lib64/python2.7/threading.py", line 952, in join
thread.join()
File "/usr/lib64/python2.7/threading.py", line 340, in wait
waiter.acquire()
KeyboardInterrupt
Why is this hanging? 这为什么挂? If it's important, I can add that
process_pdf()
runs a few subprocesses with subprocess.Popen()
. 如果是重要的,我可以添加
process_pdf()
运行的几个子进程subprocess.Popen()
Big thanks to dano for his hint. 非常感谢dano的暗示。 Fix for this issue is create queue using
Manager()
: 修复此问题的方法是使用
Manager()
创建队列:
manager = mp.Manager()
tasks, results = manager.Queue(), manager.Queue()
Edit 编辑
Tnx to ShadowRanger. Tnx到ShadowRanger。 Looks like exceptions in dispatch fixed for 2.7.10 and now we
看起来像2.7.10的调度中的例外,现在我们
can use need write wall of code for simple job :) But I didn't try it yet multiprocessing.Pool
with imap_unordered
and don't 可以使用
multiprocessing.Pool
与imap_unordered
而不是 需要为简单的工作写代码墙 :)但我还没试过
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.