Python並發性：使用apply_async（）時掛起

Question

我正在嘗試了解Python並發性。 作為實驗，我有以下程序使用進程池，並通過apply_async（）調用worker。 為了在流程（工作和結果）之間共享信息，我使用來自multiprocessing.Manager（）的隊列。

但是，當處理了工作隊列中的所有項目時，此代碼有時會掛起，我不確定為什么。 我必須運行該程序幾次才能觀察到該行為。

附帶說明一下，我可以使這項工作正確進行：我發現了一種設計模式，人們有時將其稱為“毒丸”方法，並且這種方法似乎可行。 （在我的worker（）方法中，當我的工作隊列包含一個前哨值時，我進入一個無限循環並中斷該循環。在我正在運行的進程中，在工作隊列上創建了盡可能多的前哨值）。

但是我仍然對找出此代碼為什么掛起感興趣。 我得到如下輸出（進程ID在方括號中）：

Found 8 CPUs.  
Operation queue has 20 items.  
Will start 2 processes.  
Joining pool...  
[5885] entering worker() with work_queue size of 20  
[5885] processed work item 0  
[5885] worker() still running because work_queue has size 19  
[5885] processed work item 1  
[5885] worker() still running because work_queue has size 18  
[5885] processed work item 2  
[5885] worker() still running because work_queue has size 17  
[5885] processed work item 3  
[5885] worker() still running because work_queue has size 16  
[5885] processed work item 4  
[5885] worker() still running because work_queue has size 15  
[5885] processed work item 5  
[5886] entering worker() with work_queue size of 14  
[5885] worker() still running because work_queue has size 14  
[5886] processed work item 6  
[5886] worker() still running because work_queue has size 13  
[5885] processed work item 7  
[5886] processed work item 8  
[5885] worker() still running because work_queue has size 11  
[5886] worker() still running because work_queue has size 11  
[5885] processed work item 9  
[5886] processed work item 10  
[5885] worker() still running because work_queue has size 9  
[5886] worker() still running because work_queue has size 9  
[5885] processed work item 11  
[5886] processed work item 12  
[5885] worker() still running because work_queue has size 7  
[5886] worker() still running because work_queue has size 7  
[5885] processed work item 13  
[5886] processed work item 14  
[5885] worker() still running because work_queue has size 5  
[5886] worker() still running because work_queue has size 5  
[5885] processed work item 15  
[5886] processed work item 16  
[5885] worker() still running because work_queue has size 3  
[5886] worker() still running because work_queue has size 3  
[5885] processed work item 17  
[5886] processed work item 18  
[5885] worker() still running because work_queue has size 1  
[5886] worker() still running because work_queue has size 1  
[5885] processed work item 19  
[5885] worker() still running because work_queue has size 0  
[5885] worker() is finished; returning results of size 20  

（程序掛在最后一行。另一個進程5886似乎沒有完成。）

import multiprocessing
from multiprocessing import Pool
import os

# Python 2.7.6 on Linux

# Worker (consumer) process
def worker(work_queue, results_queue):
    print "[%d] entering worker() with work_queue size of %d" % (os.getpid(), work_queue.qsize())
    while not work_queue.empty():
        item = work_queue.get()   
        print "[%d] processed work item %s" % (os.getpid(), item)  
        s = '[%d] has processed %s.' % (os.getpid(), item)
        results_queue.put(s)
        work_queue.task_done()
        print "[%d] worker() still running because work_queue has size %d" % (os.getpid(), work_queue.qsize())
    print "[%d] worker() is finished; returning results of size %d" % (os.getpid(), results_queue.qsize())

if __name__ == '__main__':

    MAX_PROCESS = 2     # Max number of processes to create
    MAX_ITEMS = 20      # Max work items to process

    m = multiprocessing.Manager()
    results_queue = m.Queue()
    work_queue = m.Queue()

    # Assign work
    for x in xrange(MAX_ITEMS):
        work_queue.put(x)     

    print "Found %d CPUs." % multiprocessing.cpu_count()
    print "Operation queue has %d items." % work_queue.qsize()
    print "Will start %d processes." % MAX_PROCESS

    # Pool method
    pool = Pool(processes=MAX_PROCESS)
    for n in range(0, MAX_PROCESS):
        pool.apply_async(worker, args=(work_queue, results_queue))
    pool.close()
    print "Joining pool..."
    pool.join()
    print "Joining pool finished..."
    print "--- After pool completion ---"

    print "Work queue has %d items." % work_queue.qsize()
    print "Results queue has %d items." % results_queue.qsize()
    print "Results are:"

    while not results_queue.empty():
        item = results_queue.get()
        print str(item)
        results_queue.task_done()
    print "--End---"

謝謝您的幫助。

Answer 1

您遇到了競爭狀況-進程5886看到Pool有一個項目：

[5886] worker() still running because work_queue has size 1

因此，它循環返回到阻塞的get調用：

while not work_queue.empty(): # It sees it's not emtpy here
    item = work_queue.get()   # But there's no item left by the time it gets here!

但是，在調用work_queue.empty()之后但在調用work_queue.get() ，其他工作work_queue.get() （ 5885 ）消耗了隊列中的最后一項：

[5885] processed work item 19  
[5885] worker() still running because work_queue has size 0  
[5885] worker() is finished; returning results of size 20  

所以，現在5886將永遠阻塞的get 。 通常，如果隊列有多個使用者，則不應使用empty()方法來決定是否進行阻塞get()調用，因為它容易受到這種競爭條件的影響。 使用您提到的“毒葯” /前哨方法是處理此情況的正確方法，或者使用無阻塞的get調用，並在發生此異常時捕獲Empty異常：

try:
    item = work_queue.get_nowait()
    print "[%d] processed work item %s" % (os.getpid(), item)  
    s = '[%d] has processed %s.' % (os.getpid(), item)
    results_queue.put(s)
    work_queue.task_done()
    print "[%d] worker() still running because work_queue has size %d" % (os.getpid(), work_queue.qsize())
except Queue.Empty:
    print "[%d] worker() is finished; returning results of size %d" % (os.getpid(), results_queue.qsize())

請注意，僅當您知道一旦工作人員開始從隊列中消耗隊列時，隊列的大小將永遠不會增長時，才可以使用此方法。 否則，您可以決定當仍有更多項目要添加到隊列時，工作人員應退出。