[英]How do you deal with an exception raised by celery (not your code)?
So in my flask app right now I am using Celery to deploy servers on remote machines. 因此,现在在我的烧瓶应用程序中,我正在使用Celery在远程计算机上部署服务器。 Right now, I have an enum, status, which indicates the lifecycle of my deployment process:
现在,我有一个枚举状态,它指示部署过程的生命周期:
@celery.task(bind=True)
def deploy_server(self, server_id):
server = Server.query.get(server_id)
if not server.can_launch():
return
try:
server.status = RemoteStatus.LAUNCHING
db.session.commit()
verify_DNS(server)
host = server.server.ssh_user + '@' + server.server.ip
execute(fabric_deploy_server, self, server, hosts=host)
server.status = RemoteStatus.LAUNCHED
db.session.commit()
except Exception as e:
server.status = RemoteStatus.ERROR
db.session.commit()
traceback.print_exc()
raise e
As you can see, when a server is being deployed, its status is changed to "Launching". 如您所见,在部署服务器时,其状态将更改为“启动”。 If there is an exception, it will be changed to ERROR.
如果有异常,它将更改为ERROR。
I found one exception which completely bypasses this bloc of code: when I overloaded my celery server with too many requests, I get this exception: 我发现了一个完全绕过该代码段的异常:当我的celery服务器因太多请求而过载时,我得到了这个异常:
[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',):
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task
uuid, retval, task_request, publish_result,
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set
return self.ensure(self._set, (key, value), **retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure
**retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set
pipe.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute
return execute(conn, stack, raise_on_error)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction
connection.send_packed_command(all_cmds)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect
self.on_connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect
if nativestr(self.read_response()) != 'OK':
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response
raise error
ConnectionError: max number of clients reached
exc, exc_info.traceback)))
My biggest problem with this is that this error is raised somewhere outside of my Try/Catch bloc. 我最大的问题是,此错误在我的“尝试/捕获”组之外的某个地方引发。 Hence, when this exception occurs, all my servers remain in the "Launching" mode rather than "Error".
因此,发生此异常时,我所有的服务器都保持“启动”模式,而不是“错误”模式。
In redis 2.4 there is a hard coded limit of max number of connections which is 10,000. 在redis 2.4中,最大连接数的硬编码限制为10,000。 In redis 2.6+ you can specify the max number of clients in redis.conf.
在redis 2.6+中,您可以在redis.conf中指定最大客户端数。 also this is not a problem of celery your broker redis refused to accept connections that's the problem.
这也不是问题,因为您的经纪人redis拒绝接受连接。
Set the max number of clients that can be handled by redis simultaneously using redis CLI. 使用redis CLI设置redis可同时处理的最大客户端数。 Check out redis clients
签出Redis客户
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.