[英]python multiprocessing callback
i have a list of functions that does some job like download html from a url(each function is very different so i can't make a single function to accept url and downlaod). 我有一个功能列表,可以完成一些工作,例如从网址下载html(每个功能都非常不同,所以我不能让一个函数接受url和downlaod)。 i have used multiprocessing to speed up the task. 我使用多处理来加速任务。 below is my code 下面是我的代码
def runInParallel(list_of_functions):
for fn in list_of_functions:
proc = [Process(target=fn[1]).start() for fn in list_of_functions]
for p in proc:
p.join()
what i want is how to store the result that each function returns? 我想要的是如何存储每个函数返回的结果? each function returns a dict that i need to parse and store in database and i dont want to repeat these steps in each function so what i want is some sort of callback that can be passed with results returned from fucntions. 每个函数返回一个我需要解析并存储在数据库中的字典,我不想在每个函数中重复这些步骤,所以我想要的是某种回调,可以通过fucntions返回的结果传递。 how can i achieve that? 我怎么能实现这一目标?
EDIT: using pool
but throws error. 编辑:使用pool
但抛出错误。 i have following for list_of_functions
: 我有以下list_of_functions
:
[('f1', <function f1 at 0x7f34c11c9ed8>), ('f2', <function f2 at 0x7f34c11c9f50>)]
def runInParallel(list_of_functions):
import multiprocessing
pool = multiprocessing.Pool(processes = 3)
x = pool.map(lambda f: f(), list_of_functions)
print x
File "main.py", line 31, in <module>
runInParallel(all_functions)
File "main.py", line 11, in runInParallel
x = pool.map(lambda f: f(), list_of_functions)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
As mentioned in the comments above: if you use Process
directly you need to set up a queue where the processes put
into, so you can get
from the parent process: 正如上述提到的意见:如果你使用Process
直接,你需要建立一个队列,其中的过程put
进去,这样你就可以get
从父进程:
from multiprocessing import Process, Queue
from time import sleep
def f1(queue):
sleep(1) # get url, "simulated" by sleep
queue.put(dict(iam="type 1"))
def f2(queue):
sleep(1.5)
queue.put(dict(iam="type 2"))
def f3(queue):
sleep(0.5)
queue.put(dict(iam="type 3"))
def runInParallel(list_of_functions):
queue = Queue()
for fn in list_of_functions:
proc = [Process(target=fn[1], args=(queue,)) for fn in list_of_functions]
for p in proc:
p.start()
res = []
for p in proc:
p.join()
res.append(queue.get())
return res
if __name__ == '__main__':
list_of_functions = [("f1", f1), ("f2", f2), ("f3", f3)]
for d in runInParallel(list_of_functions):
print d
Prints: 打印:
{'iam': 'type 3'}
{'iam': 'type f1'}
{'iam': 'type f2'}
If your functions basically do all the same (fetching urls and process the html in some way), then merging your functions into one with some if
/ elif
logic allows you to use map
and you would not need any queue: 如果你的函数基本上完全相同(获取url并以某种方式处理html),那么将你的函数合并到一个if
/ elif
逻辑允许你使用map
并且你不需要任何队列:
from multiprocessing import Pool
from time import sleep
def f(arg):
url, typ = arg
if typ == 'a':
sleep(1) # instead you would do something with `url` here
return dict(iam="type 1", url=url)
elif typ == 'b':
sleep(1.5)
return dict(iam="type 2", url=url)
elif typ == 'c':
sleep(0.5)
return dict(iam="type 3", url=url)
def runInParallel(work):
p = Pool(3)
return p.map(f, work)
if __name__ == '__main__':
work = [('http://url1', 'a'),
('http://url2', 'b'),
('http://url3', 'c'),
]
for d in runInParallel(work):
print d
Prints: 打印:
{'url': 'http://url1', 'iam': 'type 1'}
{'url': 'http://url2', 'iam': 'type 2'}
{'url': 'http://url3', 'iam': 'type 3'}
Both scripts work both on Windows as in Unix environments (tried it on OSX) 这两个脚本都可以在Windows上运行,就像在Unix环境中一样(在OSX上试过)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.