[英]Celery shutting down worker from task_success handler not working
我试图让一个工人一次只运行一个任务,然后关闭。 我已经使关机部分正常工作(这里有一些背景: celery通过在task_postrun信号中提高SystemExit来尝试关机工作者,但总是挂起,并且主进程永远不会退出 ),但是当它关机时,我得到了一个错误:
[2013-02-13 12:19:05,689: CRITICAL/MainProcess] Couldn't ack 1, reason:AttributeError("'NoneType' object has no attribute 'method_writer'",)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/kombu/transport/base.py", line 104, in ack_log_error
self.ack()
File "/usr/local/lib/python2.7/site-packages/kombu/transport/base.py", line 99, in ack
self.channel.basic_ack(self.delivery_tag)
File "/usr/local/lib/python2.7/site-packages/amqplib/client_0_8/channel.py", line 1742, in basic_ack
self._send_method((60, 80), args)
File "/usr/local/lib/python2.7/site-packages/amqplib/client_0_8/abstract_channel.py", line 75, in _send_method
self.connection.method_writer.write_method(self.channel_id,
AttributeError: 'NoneType' object has no attribute 'method_writer'
为什么会这样呢? 它不仅不应答,而且还清除了队列中剩下的所有其他任务(大问题)。
我该如何解决?
UPDATE
以下是更新了所有内容的堆栈跟踪信息(pip install -U kombu amqp amqplib celery):
[2013-02-13 11:58:05,357: CRITICAL/MainProcess] Internal error: AttributeError("'NoneType' object has no attribute 'method_writer'",)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/worker/__init__.py", line 372, in process_task
req.execute_using_pool(self.pool)
File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 219, in execute_using_pool
timeout=task.time_limit)
File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/base.py", line 137, in apply_async
**options)
File "/usr/local/lib/python2.7/dist-packages/celery/concurrency/base.py", line 27, in apply_target
callback(target(*args, **kwargs))
File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 333, in on_success
self.acknowledge()
File "/usr/local/lib/python2.7/dist-packages/celery/worker/job.py", line 439, in acknowledge
self.on_ack(logger, self.connection_errors)
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 98, in ack_log_error
self.ack()
File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 93, in ack
self.channel.basic_ack(self.delivery_tag)
File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1562, in basic_ack
self._send_method((60, 80), args)
File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 57, in _send_method
self.connection.method_writer.write_method(
AttributeError: 'NoneType' object has no attribute 'method_writer'
不建议在task_postrun中退出,因为task_postrun是在“任务主体”错误处理之外执行的。
任务调用sys.exit时发生的确切定义不明确,实际上取决于所使用的池。
通过多处理,子进程将简单地被新进程代替。 在其他池中,工作程序将关闭,但这很可能会更改,从而与多处理行为保持一致。
在任务主体外部调用exit被视为内部错误(崩溃)。
“任务主体”是在task.__call__()
执行的任何东西。
我认为也许对此更好的解决方案是使用自定义执行策略:
from celery.worker import strategy
from functools import wraps
@staticmethod
def shutdown_after_strategy(task, app, consumer):
default_handler = strategy.default(task, app, consumer)
def _shutdown_to_exit_after(fun):
@wraps(fun)
def _inner(*args, **kwargs):
try:
return fun(*args, **kwargs)
finally:
raise SystemExit()
return _inner
return _decorate_to_exit_after(default_handler)
@celery.task(Strategy=shutdown_after_strategy)
def shutdown_after():
print('will shutdown after this')
这不是很漂亮,但是执行策略可以优化任务的执行并且不易于扩展(工作人员通过缓存Task.Strategy来“预编译”每种任务类型的执行路径)
在Celery 3.1中,您可以使用“引导步骤”扩展工作人员和使用者,因此可能会有一个不错的解决方案。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.