![](/img/trans.png)
[英]Naive and easiest way to decompose independent loop into parallel threads/processes
[英]Parallel independent processes
我想循环启动多个进程,但是由于它们要花很长时间才能完成,所以我认为最好并行运行它们。 所有这些过程都是独立的,即它们并不相互依赖。 这是一个小示例,说明了我正在处理的循环类型:
inDir = '/path/to/your/dir/'
inTxtList = ['a.txt','b.txt','c.txt','d.txt','e.txt']
for i in inTxtList:
myfile = open(i,'w')
myfile.write("This is a text file written in python\n")
myfile.close()
我尝试了multiprocessing
程序包,并提供了以下代码:
import multiprocessing
def worker(num):
"""thread worker function"""
myfile = open(num,'w')
myfile.write("This is my first text file written in python\n")
myfile.close()
return
if __name__ == '__main__':
jobs = []
for i in inTxtList:
p = multiprocessing.Process(target=worker, args=(inDir+i,))
jobs.append(p)
p.start()
p.join()
它实际上在工作,但我不知道如何设置工人数。 你能帮我吗?
使用multiprocessing.Pool.map
。 创建Pool
对象时,可以通过指定processes
参数来指定工作processes
数:
import os
import multiprocessing
def worker(num):
with open(num, 'w') as f:
f.write("This is my first text file written in python\n")
if __name__ == '__main__':
number_of_workers = 4
pool = multiprocessing.Pool(processes=number_of_workers)
pool.map(worker, [os.path.join(inDir, i) for i in inTxtList])
pool.close()
pool.join()
顺便说一句,请使用os.path.join
代替手动连接路径组件。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.