[英]Python - weird behavior with Queues, Managers and Multiprocessing
在使用multiprocessing
模塊同時下載我的寵物項目時,我遇到了一個涉及由multiprocessing.Manager()
對象生成的Queue
對象的奇怪行為。
根據我將Queue對象(通過Manager
生成)放在另一個Queue
對象(也是通過Manager
生成)中的方式,我得到了一個不同的行為,據我理解,同樣的事情。 這是一個最小的工作示例:
import multiprocessing
import Queue
def work(inbound_queue, keep_going):
while keep_going.value == 1:
try:
outbound_queue = inbound_queue.get(False) # this fails in case 3
#do some work
outbound_queue.put("work done!")
except Queue.Empty:
pass #this is busy wait of course, it's just an example
class Weird:
def __init__(self):
self.manager = multiprocessing.Manager()
self.queue = self.manager.Queue()
self.keep_going = multiprocessing.Value("i", 1)
self.worker = multiprocessing.Process(target = work, args = (self.queue, self.keep_going))
self.worker.start()
def stop(self): #close and join the second process
self.keep_going.value = 0
self.worker.join()
def queueFromOutside(self, q):
self.queue.put(q)
return q
def queueFromNewManager(self):
q = multiprocessing.Manager().Queue()
self.queue.put(q)
return q
def queueFromOwnManager(self):
q = self.manager.Queue()
self.queue.put(q)
return q
if __name__ == '__main__':
instance = Weird()
# CASE 1
queue = multiprocessing.Manager().Queue()
q1 = instance.queueFromOutside(queue) # Works fine
print "1: ", q1.get()
# CASE 2
q2 = instance.queueFromNewManager() # Works fine
print "2: ", q2.get()
# CASE 3
q3 = instance.queueFromOwnManager() # Error
print "3: ", q3.get()
instance.stop() #sadly never called :(
及其輸出(python 2.7.10 x86,windows)。
主要輸出:
1: work done!
2: work done!
3:
然后工作進程崩潰,讓q3.get()掛起。
工作流程的輸出:
Process Process-2:
Traceback (most recent call last):
File "C:\Python27\lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\Python27\lib\multiprocessing\process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "J:\Dropbox\Python\queues2.py", line 7, in work
outbound_queue = inbound_queue.get(False) # this fails in case 3
File "<string>", line 2, in get
File "C:\Python27\lib\multiprocessing\managers.py", line 774, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Unserializable message: ('#RETURN', <Queue.Queue instance at 0x025A22B0>)
---------------------------------------------------------------------------
所以問題是:為什么第三種情況導致RemoteError
?
提供的示例與實際項目中的代碼結構不相似,但我確實將隊列發送到正在運行的進程,如果我使用方法#1和#2進行處理,它的工作正常。 但是使用方法#3會很好,因為它可以省去每次獲取Manager
的麻煩,這可能會花費很長的時間(我現在正在使用的機器上大約100毫秒)。
這個問題出於好奇,因為我還在學習multiprocessing
模塊中所有很酷的東西。
更新,嘗試澄清問題:在案例3( queueFromOwnManager
)中,為什么self.manager.Queue()
創建一個曾經放入self.queue
的隊列,在創建隊列時無法使用self.queue.get()
檢索with multiprocessing.Manager().Queue()
可以檢索multiprocessing.Manager().Queue()
嗎? 執行3個案件的順序無關緊要。 理想情況下,在3個示例中的3個方法調用中的任何一個之前和之后, instance.queue
將為空。
更新2:使得exaple更像我在代碼中實際做的那樣
更新了答案,我在main中添加了代碼,用於從列表ls中填充和打印放置在隊列self.queue中的項目。 我還添加了一個語句,可用於從外部函數中的ls中檢索self.queue中的項目。
import multiprocessing
import Queue
def work(inbound_queue, keep_going):
while keep_going.value == 1:
try:
pass
#outbound_queue = inbound_queue.get(False) # this fails in case 3 #<--- an error here, wherefrom does it get truthvalues?
#do some work
#outbound_queue.put("work done!")
except: #Queue.Empty: <--- self.queue.Empty(): when instantiated below
pass #this is busy wait of course, it's just an example
class Weird:
def __init__(self):
self.manager = multiprocessing.Manager()
self.queue = self.manager.Queue()
self.queue2 = multiprocessing.Manager().Queue()
self.keep_going = multiprocessing.Value("i", 1)
self.worker = multiprocessing.Process(target = work, args = (self.queue, self.keep_going))
self.worker.start()
def stop(self): #close and join the second process
self.keep_going.value = 0
#self.worker.join()
def queueFromOutside(self, q):
ls = [1,2,3,4,5]
# populate self.queue with elements from list ls
for i in ls:
self.queue.put(i)
return self.queue
def queueFromNewManager(self):
#q = multiprocessing.Manager().Queue() <---- note that you state that manager and not "queue"
# is the name of the queue in this step,
# therefor errormsg, at this step manager
# is empty and self.queue has been given ls
ls = [5,6,7,8]
# populate self.queue with elements from list ls
for i in ls:
self.queue.put(i)
return self.queue
def queueFromOwnManager(self):
q = self.manager.Queue()
ls = [5,6,7,8]
# populate self.queue with elements from list ls
for i in ls:
self.queue.put(i)
return self.queue
def wait_completion(self): #<---- function that waits until tasks are done, and joins all
# tasks as a last step, check docs how to add tasks and data to manager
"""Wait for completion of all the tasks in the queue"""
self.tasks.join()
if __name__ == '__main__':
instance = Weird()
# CASE 1
q1 = instance.queueFromOutside(instance.queue2) # Works fine
print "1: ", q1.get()
#this code gets data from instance.queue in external functions
if not instance.queue.empty():
item = instance.queue.get(True)
print item,"item"
# CASE 2
q2 = instance.queueFromNewManager() # Works fine
print "2: ", q2.get()
# CASE 3
q3 = instance.queueFromOwnManager() # Error
print "3: ", q3.get()
instance.stop() #sadly never called :(
# when all tasks done wait_completion(self) can be printed here to join all tasks
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.