多处理 - 将工作发送给工人，接收结果，更新工人，重新计算

Question

I am trying to spawn workers, give them some tasks, send results back to the main process which processes the results, update all the workers with new information and send them more tasks. I want to do this a certain amount of iterations before ending the workers.

My code seems to be working fine, but not all the workers are beeing updated for each iteration. Eg an output I received was:

Creating  4  workers
Finished workin 4
Finished workin 6
Finished workin 3
Finished workin 4

According to the Pipe Documentation recv() should block until there is something to receive. Why is this not the case in my example?

from multiprocessing import Process, Queue, JoinableQueue, Pipe, Event, Barrier, Value
import time

def worker(qu_task,qu_results,chi,ev,bar,val):
    ext=False
    dum=0
    while True:
        ev.wait()
        while qu_task.empty() is False:
            a=qu_task.get() # Pop work from queue
            if a is None: #Poison pill
                bar.wait() # Wait for all processes so that they all receive "None"
                ext=True
                qu_task.task_done() #
                print("Finished workin", dum)
                break

            # Do stuff
            time.sleep(.1)
            qu_results.put(111)
            qu_task.task_done()

#        time.sleep(1.0)

        if ext: # Break out if finished
            break

        data1, data2  = chi.recv() # Blocks until there its something to receive
        dum+=data2
        # Do stuff

def main(NProc, NScen, NTime, branch):
    qu_tasks=JoinableQueue() # To use task_done() and join()
    qu_results=Queue()
    workers=[]
    val=Value('i',0)
    conn=[]
    bar=Barrier(NProc)
    ev=Event()
    print("Creating ", NProc, " workers")
    for i in range(NProc):
        par,chi=Pipe()
        workers.append(Process(target=worker,args=(qu_tasks,qu_results,chi,ev,bar,val)))
        conn.append(par)

    for w in workers:
        w.start()

    for t in reversed(range(NTime)):
        for s in range(NScen):
            a=0.0
            for b in branch:
                k=(b,s,t)
                qu_tasks.put(k) # Send work

            ev.set() # Start workers
            qu_tasks.join() # Synchronize workers
            ev.clear() # Reset lock
            for idx,i in enumerate(branch):
                a+=qu_results.get()
            #print(a)

            kk=[1,2,3,5]
            for idx,c in enumerate(conn):
                c.send(([i*(1+idx) for i in kk],1)) # Send more data to workers

    for i in range(NProc): # Kill workers
        qu_tasks.put(None)
    ev.set()
    qu_tasks.join()

    for w in workers: # Wait for them to finish
        w.join()

if __name__=="__main__":
    branch=[i for i in range(1,7)]
    NTime=2
    NScen=3

    main(4,NScen, NTime, branch)

Answer 1

Pipe.recv() waits, do not worry about that. It just does not get invoked 6 times, as the loop bails out sooner because of the None .

Remember that asynchronous programming is well, asynchronous, and there is such schedule of your communication+processing pattern when a worker starts consuming messages from the "next" batch without reaching Pipe.recv()

If you move that time.sleep(.1) around, you can reach "Finished workin 0" too.

Also, if you slow down the (Joinable)Queue.put() loop, you can reach a situation when the task queue temporarily gets empty mid-loop, sooner than you have dispatched all the tasks, so the workers will reach the Pipe.recv() , but the main program will wait on the JoinableQueue.join()

Suggestion: pick either pipes or (joinable)queues, but do not pick both. Interlocking them is just hard, and brings no benefit.