[英]Multiprocessing in python compatible from 3 to 2.7
I have the following code runnning smoothly in Python 3, and I can't convert it to Python 2.7. 我在Python 3中顺利运行了以下代码,但无法将其转换为Python 2.7。
from multiprocessing import *
def func(path, filename, path_2):
#Things to do
for i in range(0,len(chr_names)): #len(chr_names) = 24
tuple_var.append((path, chr_names[i][0], chrom_sizes[i][0]))
cores = 4
with Pool(cores) as p:
p.starmap(func, tuple_var)
I get the following error. 我收到以下错误。
python AttributeError: __exit__
I know starmap is not supported in Python 2.7. 我知道Python 2.7不支持starmap。
What code should I use in Python 2.7? 我应该在Python 2.7中使用什么代码?
One simple approach, use a wrapper function: 一种简单的方法,使用包装函数:
def star_wrapper(args):
return func(*args)
....
with Pool(cores) as p:
p.map(star_wrapper, tuple_var)
Unless I'm misunderstanding you, it seems like you can use Pool
's map function in Python 2.6+. 除非我误解了您,否则您似乎可以在Python 2.6+中使用Pool
的map函数。 All you need is a function that can apply tuple arguments to the original function. 您需要的是一个可以将元组参数应用于原始函数的函数。 Something like: 就像是:
def pool_starmap(pool, fn, items):
def map_fn(args):
fn(*args)
return pool.map(map_fn, items)
cores = 4
with Pool(cores) as p:
pool_starmap(p, func, tuple_var)
其他答案已经介绍了如何移植starmap
,但是至于AttributeError: __exit__
错误,是由于multiprocessing.Pool
无法在Python 2.7中用作上下文管理器,因此您只需执行以下操作代替:
p = Pool(cores)
First: 第一:
In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers 在Python 2.x和3.0、3.1和3.2中,multiprocessing.Pool()对象不是上下文管理器
Have a look at this post for more info: Python Multiprocessing Lib Error (AttributeError: __exit__) 请查看此帖子以获取更多信息: Python多处理库错误(AttributeError:__exit__)
Second: 第二:
Use a helper function 使用辅助功能
Or choose one of the other options presented here: Multiprocessing with multiple arguments to function in Python 2.7 或选择此处提供的其他选项之一: 具有多个参数的多处理以在Python 2.7中起作用
Sample code: 样例代码:
from contextlib import contextmanager
from multiprocessing import *
@contextmanager
def terminating(thing):
try:
yield thing
finally:
thing.terminate()
def func(path, filename, path_2):
# Things to do
print(path)
print(filename)
print(path_2)
print('-----------------------------\n')
def helper(args):
return func(args[0], args[1], args[2])
def task():
tuple_var = []
for i in range(0, 10):
tuple_var.append(('hi_' + str(i), i, i))
with terminating(Pool(processes=2)) as p:
p.map(helper, tuple_var)
if __name__ == '__main__':
task()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.