监控Python并行进程的进度

Question

The following code returns the error:

local variable 'count' referenced before assignment.

Below are the two functions:

def run():
    ...
    global count
    count = 0
    with closing(Pool(processes=4)) as pool:
    pool.map(run_in_parallel, listOfIds)
    pool.terminate()
    ...

def run_in_parallel(id):
    count += 1
    if count % 1000 == 0:
        print('Processed %d %s' % (count, time.strftime('%x %X')))
    # Do main work in parallel

The error pops up only because of the modifying statement. ie

count +=1

I have read in several places that it is not a good practice to modify global variables in worker processes. But since the listOfIds is a huge list, I need some way of monitoring progress and printing to terminal once in a while. How is this to be done?

I have read other posts on StackOverflow and other places and none of the other questions address the exact same problem described above.

Answer 1

The error occurs most likely because you were not making the global variable count available via global count , like you did in your run() function.

Anyways, this will not work like you want it to work. Every spawned process will have it's own global count variable, which it will increment and which will always be 0 at creation time.

If you want to count it like that, you should use the example that Tom Dalton was linking for you:

import time
from multiprocessing import Process, Value

def func(val):
    for i in range(50):
        time.sleep(0.01)
        val.value += 1

if __name__ == '__main__':
    v = Value('i', 0)
    procs = [Process(target=func, args=(v,)) for i in range(10)]

    for p in procs: p.start()
    for p in procs: p.join()

    print v.value