python：理解一个简单的for循环的多处理

Question

I have never used multiprocessing module before.

Is there a way a for loop could be made into concurrent subprocesses. like

for i in xrange(10): list.append(i)

instead of sequential, make it parallel?

I tried using Queue module

q = Queue.Queue()

for i in xrange(10):
    q.put(i)


def addto(q):
    new.append(q.get(block=False))


processes = [Process(target=addto, args=(q,))]
for p in processes:
    p.start()
for p in processes:
    p.join()

And it gave out a long error, im pasting the last of it:

C:\WinPython-64bit-2.7.3.3\python-2.7.3.amd64\lib\pickle.pyc in save_global(self, obj, name, pack)
    746             raise PicklingError(
    747                 "Can't pickle %r: it's not found as %s.%s" %
--> 748                 (obj, module, name))
    749         else:
    750             if klass is not obj:

PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock

I also see this alot:

processes = [Process(target=func, args=(q,x)) for i in some iterable]

So okay there is a func(q,x) alright, and i have a map() or for loop/while going inside my function func() so why iteration in processes, again? I wouldn't want to loop the whole function using process but just make those particular loops into parallel processes. Why iterate over the target function with args? I mean when i have already q.put it?

What if I do

processes = Process(target=addto, args=(q,)).start()

Answer 1

Queue.Queue is for threadsafe queues, and thread primitives cannot be transferred to other processes. You want multiprocessing.Queue instead; simply replace

import Queue
q = Queue.Queue()

with

import multiprocessing
q = multiprocessing.Queue()

Additionally, new must be of type multiprocessing.managers.list .

However, note that you're just replicating a multiprocessing.Pool ; you can just write

import multiprocessing

new = multiprocessing.Manager().list()
def addto(val):
    new.append(val)

pool = multiprocessing.Pool()
for i in xrange(10):
    pool.apply_async(addto, (i,))
pool.close()
pool.join()
print(new)