Python's multiprocessing. The program does not finish join()

Question

I have a program which a use a lot. Therefore I wish to make it faster and trying to do multiprocessing. It kind of worked fine when I made the program use a low resolution (I am doing a power spectrum; low resolution means it will be done fast but it will not be very accurate). I got a ~2x speed up, but when doing a high resolution I terminated it before it was finish, after it had runned for a longer time than with the single processor.

My main file is something like this (I have defined f_min,f_max,df,t,f )

import multiprocessing as mp
from ast_power import power_spectrum

tasks = mp.cpu_count()
bound = mp.Queue()
res   = mp.Queue()
mint  = [mp.Process(target=power_spectrum,args=(t,f,bound,res)) for i in range(tasks)]


DF = (f_max-f_min)/tasks
for i in mint:
    i.start()
for i in range(1,tasks+1):
    a = i*f_min
    b = a+DF
    c = df
    d = 1
    bound.put([a,b,c,d])
for i in mint:
    i.join()
fr,p = [],[]
while tasks:
    frp,pp = res.get()
    frp,pp = list(frp),list(pp)
    fr += frp
    p  += pp
    tasks -= 1

And my ast_power look like this

import numpy as np
def power_spectrum(time, data, param, R='None' ):


    if R == 'None': #Normal
        f_min = param[0]
        f_max = param[1]
        df    = param[2]
        w     = param[3]
    else: # Multiprocessing
        f_min,f_max,df,w = param.get()
    freq = np.arange(f_min,f_max,df)    

    for i in xrange( len(freq) ):
        # Do the power spectrum...
        # will produce; power, alfa, beta

    if R == 'None': #Normal
        return freq,power,alfa,beta
    else: #Multiprocessing
        R.put([freq,power,alfa,beta])

Is I doing it right? I think it is very strange that it works for high df (low res.) and not for low df (high res.)

Any help is very preciated.

Answer 1

The join() call is going to block the process until it finishes. You will lose the benefit of multiprocessing if you wait for each process to finish before you run the next! Use start() to let the process run independently.