Python Multiprocessing pool.map对太多工作进程无响应

Question

first question on stack overflow so please bear with. I am looking to calculate the variance for group ratings (long numpy arrays). Running the program without parallel processing works fine, but given each process can run independently and there are 32 groups I am looking to make use of multiprocessing to speed things up. This works OK for small numbers of groups < 10, but after this the program will often just seemingly stop running with no error messages at an unspecified number of groups ( usually between 20 and 30 ) although less frequently will run all the way through. The arrays are quite large ( 21451 x 11462 user item ratings) and so I am wondering if the problem is caused by not enough memory, although no error messages are printed.

import numpy as np
from functools import partial
import multiprocessing

def variance_parallel(extra_matrices, group_num):
    # do some variation calculation
    # print confirmation that we have entered function, and group number
    return single_group_var

def variance(extra_matrices, num_groups):
    variance_partial = partial(variance_parallel, extra_matrices)
    for g in list(range(num_groups)):
        group_var = pool.map(variance_partial,range(g))
    return(group_var)     

num_cores = multiprocessing.cpu_count() - 1
pool = multiprocessing.Pool(processes=num_cores)
variance(extra_matrices, num_groups)

Running the above code shows the program progressively building the number of groups it is checking variance on ([0],[0,1],[0,1,2],...) before eventually printing nothing.

Thanks in advance for any help and apologies if my formatting / question is a bit off!

Answer 1

• Multiple processes do not share data
• Data sent to processes needs to be copied

Since the arrays are large, the issue is very likely to do with said copying of large arrays to the processes. Further more in Python's multiprocessing, sending data to processes is done by serialisation which is (a) CPU intensive and (b) takes extra memory in and by it self.

In short multi processing is not a good fit for your use case. Since numpy is a native code extension (where GIL does not apply) and is thread safe, best to use threading instead of multiprocessing. With threading, the worker threads can share data via their parent process's address space which makes away with having to copy.

That should stop the program from running out of memory.

However, for threads to share address space the data they share needs to be bound to an object, like in a python class.

Something like the below - untested as the code sample is incomplete.

import numpy as np
from functools import partial
from threading import Thread
from multiprocessing import cpu_count

class Variance(Thread):

    def __init__(self, extra_matrices, group_num):
        Thread.__init__(self)
        self.extra_matrices = extra_matrices
        self.group_num = group_num
        self.output = None

    def run(self):
        # do some variation calculation
        # print confirmation that we have entered function, and group number
        self.output = single_group_var

num_cores = cpu_count() - 1
results = []
for g in list(range(num_groups)):
    workers = [Variance(extra_matrices, range(g)) 
               for _ in range(num_cores)]
    # Start threads
    for worker in workers:
        worker.start()
    # Wait for completion
    for worker in workers:
        worker.join()
    results.extend([w.output for w in workers])
print results