[英]Celery Task Chain and Accessing **kwargs
I have a situation similar to the one outlined here , except that instead of chaining tasks with multiple arguments, I want to chain tasks that return a dictionary with multiple entries. 我有类似于此处概述的情况,除了不是使用多个参数链接任务,我想链接返回具有多个条目的字典的任务。
This is -- very loosely and abstractly --- what I'm trying to do: 这是 - 非常松散和抽象 - 我正在努力做的事情:
tasks.py tasks.py
@task()
def task1(item1=None, item2=None):
item3 = #do some stuff with item1 and item2 to yield item3
return_object = dict(item1=item1, item2=item2, item3=item3)
return return_object
def task2(item1=None, item2=None, item3=None):
item4 = #do something with item1, item2, item3 to yield item4
return_object = dict(item1=item1, item2=item2, item3=item3, item4=item4)
return return_object
Working from ipython, I am able to call task1 individually and asynchronously, with no problems. 从ipython开始,我可以单独和异步地调用task1,没有任何问题。
I can ALSO call task2 individually with the result returned by task1 as a double-star argument: 我也可以单独调用task2,task1返回的结果为双星参数:
>>res1 = task1.s(item1=something, item2=something_else).apply_async()
>>res1.status
'SUCCESS'
>>res2 = task2.s(**res1.result).apply_async()
>>res2.status
'SUCCESS
However, what I ultimately want achieve is the same end result as above, but via a chain, and here, I can't figure out how to have task2 instantiated not with (positional) arguments returned by task1, but with task1.result as **kwargs: 但是,我最终想要实现的是与上面相同的最终结果,但是通过链,在这里,我无法弄清楚如何将task2实例化而不是使用task1返回的(位置) 参数 ,而是使用task1.result作为** kwargs:
chain_result = (task1.s(item1=something, item2=something_else) | task2.s()).apply_async() #THIS DOESN'T WORK!
I suspect that I can go back and rewrite my tasks so that they return positional arguments instead of a dictionary, and this may clear things up, but it seems to me that there should be some way to access task1's return object in task2 with the equivalent functionality of the **double star. 我怀疑我可以回去重写我的任务,这样他们就可以返回位置参数而不是字典,这可能会让事情变得清晰,但在我看来应该有办法在task2中使用等效的方法访问task1的返回对象。 **双星的功能。 I also suspect that I'm missing something here fairly obvious about either Celery subtask implementation or *args vs. **kwargs. 我也怀疑我在这里遗漏了一些关于Celery子任务实现或* args与** kwargs相当明显的东西。
Hope this makes sense. 希望这是有道理的。 And thanks in advance for any tips. 并提前感谢任何提示。
This is my take at the problem, using an abstract task class: 这是我对问题的看法,使用抽象的任务类:
from __future__ import absolute_import
from celery import Task
from myapp.tasks.celery import app
class ChainedTask(Task):
abstract = True
def __call__(self, *args, **kwargs):
if len(args) == 1 and isinstance(args[0], dict):
kwargs.update(args[0])
args = ()
return super(ChainedTask, self).__call__(*args, **kwargs)
@app.task(base=ChainedTask)
def task1(x, y):
return {'x': x * 2, 'y': y * 2, 'z': x * y}
@app.task(base=ChainedTask)
def task2(x, y, z):
return {'x': x * 3, 'y': y * 3, 'z': z * 2}
You can now define and execute your chain as such: 您现在可以定义和执行您的链:
from celery import chain
pipe = chain(task1.s(x=1, y=2) | task2.s())
pipe.apply_async()
chain
and the other canvas primitives are in the family of functional utilities like map
and reduce
. chain
和其他canvas原语属于map
和reduce
等功能实用程序。
Eg where map(target, items)
calls target(item)
for every item in the list, Python has a rarely used version of map called itertools.starmap
, which instead calls target(*item)
. 例如, map(target, items)
调用列表中每个项目的target(item)
,Python有一个很少使用的地图版本,名为itertools.starmap
,它调用target(*item)
。
While we could add starchain
and even kwstarchain
to the toolbox, these would be very specialized and probably not used as often. 虽然我们可以将starchain
甚至kwstarchain
添加到工具箱中,但这些都非常专业,可能不经常使用。
Interestingly Python has made these unnecessary with the list and generator expressions, so that map is replaced with [target(item) for item in item]
and starmap with [target(*item) for item in item]
. 有趣的是,Python使用列表和生成器表达式使这些变得不必要,因此地图被替换为[target(item) for item in item]
和星形图与[target(*item) for item in item]
[target(item) for item in item]
[target(*item) for item in item]
。
So instead of implementing several alternatives for each primitive I think we should focus on finding a more flexible way of supporting this, eg like having celery powered generator expressions (if possible, and if not something similarly powerful) 因此,我认为我们应该专注于寻找一种更灵活的方式来支持这一点,而不是为每个原语实现几个替代方案,例如像芹菜动力发电机表达式(如果可能的话,如果不是同样强大的东西)
Since this isn't built into celery, I wrote a decorator function to something similar myself. 由于这不是内置于芹菜,我自己写了一个装饰函数给类似的东西。
# Use this wrapper with functions in chains that return a tuple. The
# next function in the chain will get called with that the contents of
# tuple as (first) positional args, rather than just as just the first
# arg. Note that both the sending and receiving function must have
# this wrapper, which goes between the @task decorator and the
# function definition. This wrapper should not otherwise interfere
# when these conditions are not met.
class UnwrapMe(object):
def __init__(self, contents):
self.contents = contents
def __call__(self):
return self.contents
def wrap_for_chain(f):
""" Too much deep magic. """
@functools.wraps(f)
def _wrapper(*args, **kwargs):
if type(args[0]) == UnwrapMe:
args = list(args[0]()) + list(args[1:])
result = f(*args, **kwargs)
if type(result) == tuple and current_task.request.callbacks:
return UnwrapMe(result)
else:
return result
return _wrapper
Mine unwraps like the starchain
concept, but you could easily modify it to unwrap kwargs instead. 我想像星starchain
概念那样starchain
,但是你可以轻松修改它以解开kwargs。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.