[英]Python: Multiprocessing Does Not Complete Jobs
I'm using python 2.7 with multiprocessing::Pool to run a job in parallel 我正在使用带有multiprocessing :: Pool的python 2.7来并行运行作业
I've simplified the example below, but here's the main gist of it. 我简化了下面的例子,但这是它的主要要点。
It will create a file for each person in my dict using the apply_async()
function. 它将使用apply_async()
函数为我的dict中的每个人创建一个文件。 However when I check to see if the file was created properly, I notice that sometimes the file was not created. 但是,当我检查文件是否正确创建时,我注意到有时文件未创建。
Now I'm thinking I've done something wrong in how I used multiprocessing::Pool 现在我想我在使用多处理:: Pool方面做错了什么
Any advice? 有什么建议?
import os
from multiprocessing import Pool
def outputFile(person):
ofh=open(person+'.txt','w')
ofh.write('test\n')
ofh.close()
pool = Pool(processes=4)
for person in person_dict:
pool.apply_async(outputFile,args(person))
pool.close()
pool.join()
for person in person_dict:
print os.path.isfile(person+'.txt')
True
True
False
True
If you do not catch exceptions in sub-processes and print them yourself, you will not see them. 如果您没有捕获子流程中的异常并自行打印,您将看不到它们。 The following program produces no output: 以下程序不产生输出:
import os
from multiprocessing import Pool
def outputFile(person):
raise Exception("An exception")
pool = Pool(processes=4)
for person in range(100):
pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()
You need to catch all exceptions and manually print the traceback: 您需要捕获所有异常并手动打印回溯:
import os
from multiprocessing import Pool, Lock
import traceback
print_lock = Lock()
def outputFile(person):
try:
raise Exception("An exception")
except:
with print_lock:
print "%s: An exception occurred" % person
print traceback.format_exc()
pool = Pool(processes=4)
for person in range(100):
args = (person, print_lock)
pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()
Output 产量
0: An exception occurred
Traceback (most recent call last):
File "person.py", line 9, in outputFile
raise Exception("An exception")
Exception: An exception
1: An exception occurred
Traceback (most recent call last):
File "person.py", line 9, in outputFile
raise Exception("An exception")
Exception: An exception
...
99: An exception occurred
Traceback (most recent call last):
File "person.py", line 9, in outputFile
raise Exception("An exception")
Exception: An exception
Note: The print_lock
is used to keep the output from being interleaved. 注意: print_lock
用于保持输出交错。
Could this be related to contents of the person_dict? 这可能与person_dict的内容有关吗?
I have modified your code and run it several times. 我修改了你的代码并运行了几次。 They all produced the expected results. 他们都产生了预期的结果。
Here is the code I modified and tested: 这是我修改和测试的代码:
import os
from multiprocessing import Pool
def outputfile(person):
with open(person+'.txt','w') as ofh:
ofh.write('test\n')
person_dict = {'a': 'a', 'b': 'b', 'c':'c', 'd':'d'}
pool = Pool(processes=4)
for person in person_dict:
pool.apply_async(outputfile, (person))
pool.close()
pool.join()
for person in person_dict:
print(os.path.isfile(person+'.txt'))
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.