内部多重处理

Question

我发现这里的帖子似乎有用：

http://eli.thegreenplace.net/2012/01/16/python-parallelizing-cpu-bound-tasks-with-multiprocessing/

而且我已经尝试了使用多处理的这段代码，但是对我来说不起作用。 我对原始文件所做的唯一更改是变量out_q = queue.Queue，而不是out_q = Queue。

我相信这段代码是用python 2.x编写的，我正在使用python 3.4.2

我当然已经导入了所有必需品。

def mp_factorizer(nums, nprocs):
    def worker(nums, out_q):
        """ The worker function, invoked in a process. 'nums' is a
            list of numbers to factor. The results are placed in
            a dictionary that's pushed to a queue.
        """
        outdict = {}
        for n in nums:
            outdict[n] = factorize_naive(n)
        out_q.put(outdict)

    # Each process will get 'chunksize' nums and a queue to put his out
    # dict into
    out_q = queue.Queue()
    chunksize = int(math.ceil(len(nums) / float(nprocs)))
    procs = []

    for i in range(nprocs):
        p = multiprocessing.Process(
                target=worker,
                args=(nums[chunksize * i:chunksize * (i + 1)],
                      out_q))
        procs.append(p)
        p.start()

    # Collect all results into a single result dict. We know how many dicts
    # with results to expect.
    resultdict = {}
    for i in range(nprocs):
        resultdict.update(out_q.get())

    # Wait for all worker processes to finish
    for p in procs:
        p.join()

    return resultdict

我得到的错误是

File "E:\....\file.py", line 109, in <module>
    print (mp_factorizer(range(100000),1))
File "E:\....\file.py", line 88, in mp_factorizer
    p.start()
File "E:\...\Python\Install\lib\multiprocessing\process.py", line 105, in start
    self._popen = self.+Popen(self)
File "E:\...\Python\Install\lib\multiprocessing\context.py", line 212, in _Popen
    return default_context.get_context().Process._Popen(process_obj)
File "E:\...\Python\Install\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
File "E:\...\Python\Install\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
File "E:\...\Python\Install\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class '_thread.lock'>: attribute lookup lock on _thread failed

Answer 1

这是一个完整的程序，可以在Python 3.5.1下运行（但在Python 2下也可以正常工作）。 这样一来，您可以看到所有需要完成的工作。 它实际上并没有分解任何因素-工人的胆量被非常简单的代码取代，因为在最低级别完成的计算与您遇到的任何问题都不相关：

import multiprocessing as mp
import math

def worker(nums, out_q):
    outdict = {}
    for n in nums:
        outdict[n] = 2 * n
    out_q.put(outdict)

def mp_factorizer(nums, nprocs):
    out_q = mp.Queue()
    chunksize = int(math.ceil(len(nums) / float(nprocs)))
    procs = []

    for i in range(nprocs):
        p = mp.Process(
                target=worker,
                args=(nums[chunksize * i:chunksize * (i + 1)],
                      out_q))
        procs.append(p)
        p.start()

    resultdict = {}
    for i in range(nprocs):
        resultdict.update(out_q.get())

    for p in procs:
        p.join()

    return resultdict

if __name__ == "__main__":
    print(mp_factorizer(range(100), 3))