Problem with multiprocessing.Pool in Python

Question

I am having a bit of trouble with Parallel processing in Python. I am completely new to the concept of Parallel computing. I use multiprocessing that comes with standard Python.

I have 12 threads in my computer. I ask for 12 workers, but I am not always able to get all workers I ask for. My problem arises when I don't get access to as many workers I need to process nTasks number of tasks in my code below (currently set to four). What happens then is just that the code gets stuck and never gets to what is under the comment "# Get Results". It seems random how many workers I get (I always ask for 12), but the problem arises when I get three workers or less in the following code:

import multiprocessing as mp
import scipy as sp
import scipy.stats as spstat
import pylab

def testfunc(x0, N):
   print 'working with x0 = %s' % x0
   x = [x0]
   for i in xrange(1,N):
       x.append(spstat.norm.rvs(size = 1)) # stupid appending to make it slower
       if i % 10000 == 0:
          print 'x0 = %s, i = %s' % (x0, i)
   return sp.array(x)

def testfuncParallel(fargs):
    return testfunc(*fargs)

pool = mp.Pool(12) # I have 12 threads

nTasks = 4
N = 100000

tasks = [(x, n) for x, n in enumerate(nTasks*[N])] # nTasks different tasks

result = pool.map(testfuncParallel, tasks)
pool.close()
pool.join()

# Get results:
sim = sp.zeros((N, nTasks)) 

for nn, res in enumerate(result):    
    sim[:, nn] = res

pylab.figure()
for i in xrange(nTasks):
    pylab.subplot(nTasks,1, i + 1)
    pylab.plot(sim[:, i])

pylab.show()

I have tried to use pool.map_async instead of pool.map but I cannot get around the problem.

Thanks in advance,

Sincerely,

Matias

Answer 1

It turned out that this was only a problem when running the Debug mode in the Wing Editor, see Stephan Deibels comment in my other question multiprocessing.Pool seems to work in Windows but not in ubuntu? .