Can someone help me with parallel processing for Python?

Question

I am searching for a possible solution to read and trigger data (in my case waveform data) parallel.

I have a list (wfpathlist), just a list containing strings with given paths to the files:

for fn in wfpathlist:
    st = readWaveform(fn, bpfreq=[5.,30.])
    coinc = CoincidenceTimes(st)
    triggerlist += coinc.getCoincTimes()
    events += cutWithList(wfpath=fn, trigg=coinc.getCoincTimes(), 
    station=self.getStation())

With readWaveform just being a short function to read and filter the data:

def readWaveform(wfpath, bpfreq=[5., 30.]):
    st = read(wfpath)
    st.filter('bandpass', freqmin=bpfreq[0], freqmax=bpfreq[1])
    return st

Is there a easy solution for somebody with just a very basic understanding of programming to let this for-loop run parallel?

Thanks alot,

Dennis

Answer 1

You can implement your for-loop in a function, and call this function with the multiprocessing.Pool() object. This will parallelize the execution of your loop and should add a nice speedup.

Example :

from multiprocessing import Pool

def func(fn): 
    st = readWaveform(fn, bpfreq=[5.,30.])
    coinc = CoincidenceTimes(st)
    triggerlist += coinc.getCoincTimes()
    events += cutWithList(wfpath=fn, trigg=coinc.getCoincTimes(), 
    station=self.getStation())

if __name__ == '__main__':
    p = Pool(4) # if you have 4 cores in your processor
    p.map(func, wfpathlist)

source : https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

Note that some python-implemented function already use multiprocessing (common in numpy), so you should check your processor activity when your script is running before implementing this solution.

EDIT : my bad, you should send directly your list to pool. I corrected the code.