[英]Pipes are getting stuck--no other solution on stack overflow working
(更新)我正在構建一個模塊來分發基於代理的模型,其思想是將模型拆分到多個進程中,然后當代理到達邊界時,將它們傳遞給處理該區域的處理器。 我可以建立進程並在不進行通信的情況下工作,但是無法使數據通過管道傳遞並更新其他處理器上的模型段。
我已經在stackoverflow上嘗試了解決方案,並構建了該模型的簡單版本。 一旦將模型對象放入管道,模型就會掛起(它適用於python標准數據類型)。 簡單版本只是來回傳遞代理。
from pathos.multiprocessing import ProcessPool
from pathos.helpers import mp
import copy
class TestAgent:
"Agent Class-- Schedule iterates through each agent and \
executes step function"
def __init__(self, unique_id, model):
self.unique_id = unique_id
self.model = model
self.type = "agent"
def step(self):
pass
#print (' ', self.unique_id, "I have stepped")
class TestModel:
"Model Class iterates through schedule and executes step function for \
each agent"
def __init__(self):
self.schedule = []
self.pipe = None
self.process = None
for i in range(1000):
a = TestAgent(i, self)
self.schedule.append(a)
def step(self):
for a in self.schedule:
a.step()
if __name__ == '__main__':
pool = ProcessPool(nodes=2)
#create instance of model
test_model = TestModel()
#create copies of model to be run on 2 processors
test1 = copy.deepcopy(test_model)
#clear schedule
test1.schedule = []
#Put in only half the schedule
for i in range(0,500):
test1.schedule.append(test_model.schedule[i])
#Give process tracker number
test1.process = 1
#repeat for other processor
test2= copy.deepcopy(test_model)
test2.schedule = []
for i in range(500,1000):
test2.schedule.append(test_model.schedule[i])
test2.process = 2
#create pipe
end1, end2 = mp.Pipe()
#Main run function for each process
def run(model, pipe):
for i in range(5):
print (model.process)#, [a.unique_id for a in model.schedule])
model.step() # IT HANGS AFTER INITIAL STEP
print ("send")
pipe.send(model.schedule)
print ("closed")
sched = pipe.recv()
print ("received")
model.schedule = sched
pool.map(run, [test1, test2], [end1,end2])
代理應切換處理器並執行其打印功能。 (我的下一個問題將是同步處理器,以使它們保持在每個步驟上,但一次執行一次。)
我知道了。 我超出了python中的管道緩沖區限制(8192)。 如果代理擁有模型的副本作為屬性,則尤其如此。 下面是上面代碼的有效版本,該版本一次通過代理。 它使用Pympler來獲取所有代理的大小。
from pathos.multiprocessing import ProcessPool
from pathos.helpers import mp
import copy
# do a blocking map on the chosen function
class TestAgent:
"Agent Class-- Schedule iterates through each agent and \
executes step function"
def __init__(self, unique_id, model):
self.unique_id = unique_id
self.type = "agent"
def step(self):
pass
class TestModel:
"Model Class iterates through schedule and executes step function for \
each agent"
def __init__(self):
from pympler import asizeof
self.schedule = []
self.pipe = None
self.process = None
self.size = asizeof.asizeof
for i in range(1000):
a = TestAgent(i, self)
self.schedule.append(a)
def step(self):
for a in self.schedule:
a.step()
if __name__ == '__main__':
pool = ProcessPool(nodes=2)
#create instance of model
test_model = TestModel()
#create copies of model to be run on 2 processors
test1 = copy.deepcopy(test_model)
#clear schedule
test1.schedule = []
#Put in only half the schedule
for i in range(0,500):
test1.schedule.append(test_model.schedule[i])
#Give process tracker number
test1.process = 1
#repeat for other processor
test2= copy.deepcopy(test_model)
test2.schedule = []
for i in range(500,1000):
test2.schedule.append(test_model.schedule[i])
test2.process = 2
#create pipe
end1, end2 = mp.Pipe()
#Main run function for each process
def run(model, pipe):
for i in range(5):
agents = []
print (model.process, model.size(model.schedule) )
model.step() # IT HANGS AFTER INITIAL STEP
#agent_num = list(model.schedule._agents.keys())
for agent in model.schedule[:]:
model.schedule.remove(agent)
pipe.send(agent)
agent = pipe.recv()
agents.append(agent)
print (model.process, "all agents received")
for agent in agents:
model.schedule.append(agent)
print (model.process, len(model.schedule))
pool.map(run, [test1, test2], [end1,end2])
Mike McKerns和Thomas Moreau-謝謝您的幫助,使我走上了正確的道路。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.