[英]Python : Parallel execution of function
I would like to execute set of tasks in parallel.我想并行执行一组任务。 I have defined a function in a class which takes the parameter and executes the operation based on parameter.我在一个类中定义了一个函数,它接受参数并根据参数执行操作。 Class structure is like below.类结构如下。
from threading import Thread
from concurrent.futures import *
class Test(object):
def process_dataframe(self,id:int):
print(id*id)
def run_task(self):
thd = []
for i in range(1,10):
thd.append( "self.process_dataframe({0})".format(i))
self.run_functions_in_parallel(thd)
def run_functions_in_parallel(self,fns)->bool:
def wrap_function(self,fnToCall):
try:
eval(fnToCall)
return ("0")
except Exception as e:
return "{0}".format(e)
thd = []
isError = False
executor = ThreadPoolExecutor(max_workers=len(fns))
errorMessage = ""
for fn in fns:
t = executor.submit(wrap_function,self,fn)
thd.append(t)
for td in thd:
ret = td.result()
if ret != "0":
isError = True
errorMessage = errorMessage + """
""" + ret
if isError == True:
print (errorMessage)
raise Exception (errorMessage)
else:
return True
d=Test()
d.run_task()
I have managed to make it work and tasks are executing properly.我已经设法让它工作并且任务正在正确执行。 I am wondering whether there is better/simpler way to accomplish the same.我想知道是否有更好/更简单的方法来完成同样的任务。 I would like to keep run_functions_in_parallel method generic so that it can be used as common method in a module.我想保持 run_functions_in_parallel 方法通用,以便它可以用作模块中的通用方法。
You don't need to use a wrapper, since ThreadPoolExecutor
catches errors in a better way.您不需要使用包装器,因为ThreadPoolExecutor
以更好的方式捕获错误。 A function, that always returns True
or raises an error, don't need a return value, but if you have functions with return values, you want to call in parallel, you should return their results.一个总是返回True
或引发错误的函数不需要返回值,但如果你有返回值的函数,你想并行调用,你应该返回它们的结果。 It is a bad idea to use a magic string as indicator for errors.使用魔术字符串作为错误指示符是一个坏主意。 format(e)
of a KeyError: 0
also leads to "0"
. KeyError: 0
format(e)
也会导致"0"
。 Better use a unique value, like None
in our case.最好使用唯一值,例如在我们的示例中为None
。 Don't use eval
if you don't have to.如果不需要,请不要使用eval
。 In your case, you can use partial
.在您的情况下,您可以使用partial
。 Don't use a to large value for max_workers
.不要对max_workers
使用太大的值。
from functools import partial
from concurrent.futures import ThreadPoolExecutor
class Test(object):
def process_dataframe(self, id):
print(id*id)
def run_task(self):
functions = []
for i in range(1,10):
functions.append(partial(self.process_dataframe, i))
self.run_functions_in_parallel(functions)
def run_functions_in_parallel(self, functions, max_workers=8):
executor = ThreadPoolExecutor(max_workers=max_workers)
futures = [
executor.submit(function)
for function in functions
]
errors = []
results = []
for future in futures:
try:
result = future.result()
except Exception as e:
errors.append(e)
else:
results.append(result)
if errors:
raise Exception(errors)
return results
d = Test()
d.run_task()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.