[英]Can someone help me with parallel processing for Python?
I am searching for a possible solution to read and trigger data (in my case waveform data) parallel. 我正在寻找一种可能的解决方案,以并行读取和触发数据(在我的情况下为波形数据)。
I have a list (wfpathlist), just a list containing strings with given paths to the files: 我有一个列表(wfpathlist),只是一个列表,其中包含具有给定文件路径的字符串:
for fn in wfpathlist:
st = readWaveform(fn, bpfreq=[5.,30.])
coinc = CoincidenceTimes(st)
triggerlist += coinc.getCoincTimes()
events += cutWithList(wfpath=fn, trigg=coinc.getCoincTimes(),
station=self.getStation())
With readWaveform just being a short function to read and filter the data: readWaveform只是读取和过滤数据的简短功能:
def readWaveform(wfpath, bpfreq=[5., 30.]):
st = read(wfpath)
st.filter('bandpass', freqmin=bpfreq[0], freqmax=bpfreq[1])
return st
Is there a easy solution for somebody with just a very basic understanding of programming to let this for-loop run parallel? 对于只有非常基本的编程知识的人,有没有一种简单的解决方案可以让此for循环并行运行?
Thanks alot, 非常感谢,
Dennis 丹尼斯
You can implement your for-loop in a function, and call this function with the multiprocessing.Pool() object. 您可以在函数中实现for循环,然后使用multiprocessing.Pool()对象调用此函数。 This will parallelize the execution of your loop and should add a nice speedup. 这将并行化循环的执行,并应增加不错的加速比。
Example : 范例:
from multiprocessing import Pool
def func(fn):
st = readWaveform(fn, bpfreq=[5.,30.])
coinc = CoincidenceTimes(st)
triggerlist += coinc.getCoincTimes()
events += cutWithList(wfpath=fn, trigg=coinc.getCoincTimes(),
station=self.getStation())
if __name__ == '__main__':
p = Pool(4) # if you have 4 cores in your processor
p.map(func, wfpathlist)
source : https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers 来源: https : //docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers
Note that some python-implemented function already use multiprocessing (common in numpy), so you should check your processor activity when your script is running before implementing this solution. 请注意,一些由python实现的函数已经使用了多重处理(在numpy中很常见),因此在实现此解决方案之前,应在脚本运行时检查处理器的活动。
EDIT : my bad, you should send directly your list to pool. 编辑:我不好,您应该直接将列表发送到池中。 I corrected the code. 我更正了代码。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.