Python：多處理無法完成作業

Question

我正在使用帶有multiprocessing :: Pool的python 2.7來並行運行作業

我簡化了下面的例子，但這是它的主要要點。

它將使用apply_async()函數為我的dict中的每個人創建一個文件。 但是，當我檢查文件是否正確創建時，我注意到有時文件未創建。

現在我想我在使用多處理:: Pool方面做錯了什么

有什么建議？

import os
from multiprocessing import Pool

def outputFile(person):
    ofh=open(person+'.txt','w')
    ofh.write('test\n')
    ofh.close()

pool = Pool(processes=4)
for person in person_dict:
    pool.apply_async(outputFile,args(person))
pool.close()
pool.join()
for person in person_dict:
    print os.path.isfile(person+'.txt')

---

True
True
False
True

Answer 1

如果您沒有捕獲子流程中的異常並自行打印，您將看不到它們。 以下程序不產生輸出：

import os
from multiprocessing import Pool

def outputFile(person):
    raise Exception("An exception")

pool = Pool(processes=4)
for person in range(100):
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

您需要捕獲所有異常並手動打印回溯：

import os
from multiprocessing import Pool, Lock
import traceback

print_lock = Lock()

def outputFile(person):
    try:
        raise Exception("An exception")
    except:
        with print_lock:
            print "%s: An exception occurred" % person
            print traceback.format_exc()

pool = Pool(processes=4)
for person in range(100):
    args = (person, print_lock)
    pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()

產量

0: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

1: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

...

99: An exception occurred
Traceback (most recent call last):
  File "person.py", line 9, in outputFile
    raise Exception("An exception")
Exception: An exception

注意： print_lock用於保持輸出交錯。

Answer 2

這可能與person_dict的內容有關嗎？

我修改了你的代碼並運行了幾次。 他們都產生了預期的結果。

這是我修改和測試的代碼：

import os
from multiprocessing import Pool

def outputfile(person):
    with open(person+'.txt','w') as ofh:
        ofh.write('test\n')

person_dict = {'a': 'a', 'b': 'b', 'c':'c', 'd':'d'}

pool = Pool(processes=4)
for person in person_dict:
    pool.apply_async(outputfile, (person))
pool.close()
pool.join()

for person in person_dict:
    print(os.path.isfile(person+'.txt'))