ProcessPoolExecutor 开销？？ 对于大型矩阵运算，并行处理比单个处理花费更多时间

Question

My python code contains a numpy dot operation of huge size (over 2^(tens...)) matrix and vector. To reduce the computing time, I applied parallel processing by dividing the matrix suited for the number of cpu cores. I used concurrent.futures.ProcessPoolExecutor . My issue is that the parallel processing takes much more time than single processing.

The following is my code.

1. single process code.

self._vector = np.dot(matrix, self._vector)

2. parallel processing code.

    each_worker_coverage = int(self._dimension/self.workers)
         args = []
         for i in range(self.workers):
             if (i+1)*each_worker_coverage < self._dimension:
                 arg = [i, gate[i*each_worker_coverage:(i+1)*each_worker_coverage], self._vector]
             else:
                 arg = [i, gate[i*each_worker_coverage:self._dimension], self._vector]
             args.append(arg)
         pool = futures.ProcessPoolExecutor(self.workers)
         results = list(pool.map(innerproduct, args, chunksize=1))
         for result in results:
             if (result[0]+1)*each_worker_coverage < self._dimension:
                 self._vector[result[0]*each_worker_coverage:(result[0]+1)*each_worker_coverage] = result[1]
             else:
                 self._vector[result[0]*each_worker_coverage:self._dimension] = result[1]

The innerproduct function called in parallel is as follows.

def innerproduct(args):
    answer = np.dot(args[1], args[2])
    return args[0], answer
    ```

For a 2^14 x 2^14 matrix and a 2^14 vector, the single process code takes only 0.05 seconds, but the parallel processing code takes 6.2 seconds.
I also checked the time with the `innerproduct` method, and it only takes 0.02~0.03 sec.
I don't understand this situation.
Why does the parallel processing (multi-processing not multi-threading) take more time?

Answer 1

To exactly know the cause of the slowdown you would have to measure how long everything takes, and with multiprocessing and multithreading that can be tricky.

So what follows is my best guess. For multiprocessing to work, the parent process has to transfer the data used in the calculations to the worker processes. The time this takes depends on the amount of data. Transferring a 2^14 x 2^14 matrix is probably going to take a significant amount of time. I suspect that this data transfer is what is taking the extra time.

If you are using an operating system that uses the fork startmethod for multiprocessing / concurrent.futures , there is a way around this data transfer . These operating systems are for example Linux, *BSD and macOS (but not ms-windows).

On the abovementioned operating systems, multiprocessing uses the fork system call to create its workers. This system call creates a copy of the parent process as the child processes. So if you create the vectors and matrices before creating the ProcessPoolExecutor , the workers will inherit that data. This is not a very costly or time consuming operation because all these OS's use copy-on-write for managing memory pages. As long as the original matrix isn't changed, all programs using it are reading from the same memory pages. This inheriting of the data means you don't have to pass the data explicitly to the worker. You just have to pass a small data structure that says on which index ranges a worker has to operate.

Unfortunately, due to technical limitations of the platform, this doesn't work on ms-windows. What you could do on ms-windows is store the original matrix and vector memory mapped binary files before you create the Executor . If you tag these mappings with a name, the worker processes should be able to map the same data into their memory without having to transfer them. I think is it possible to instruct numpy to use such a raw binary array without recreating it.

On both platforms you could use the same technique to "send data back" to the parent process; save the data in shared memory and return the filename or tagname to the parent process.

Answer 2

If you are using modern versions of NumPy and OS, it's most likely that

self._vector = np.dot(matrix, self._vector)

is already optimized and uses all your CPU cores.

If np.show_config() displays openblas or MKL you may run a simple test:

a = np.random.rand(7000, 7000)
b = np.random.rand(7000, 7000)
np.dot(a, b)

It should use all CPU cores for a couple of seconds.

If it's not, you may install OpenBLAS or MKL and reinstall NumPy. See Using MKL to boost Numpy performance on Ubuntu