How do you deal with an exception raised by celery (not your code)?

Question

So in my flask app right now I am using Celery to deploy servers on remote machines. Right now, I have an enum, status, which indicates the lifecycle of my deployment process:

@celery.task(bind=True)
def deploy_server(self, server_id):
  server = Server.query.get(server_id)
  if not server.can_launch():
    return

  try:
    server.status = RemoteStatus.LAUNCHING
    db.session.commit()

    verify_DNS(server)

    host = server.server.ssh_user + '@' + server.server.ip
    execute(fabric_deploy_server, self, server, hosts=host)

    server.status = RemoteStatus.LAUNCHED
    db.session.commit()
  except Exception as e:
    server.status = RemoteStatus.ERROR
    db.session.commit()
    traceback.print_exc()
    raise e

As you can see, when a server is being deployed, its status is changed to "Launching". If there is an exception, it will be changed to ERROR.

I found one exception which completely bypasses this bloc of code: when I overloaded my celery server with too many requests, I get this exception:

[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',): 
Traceback (most recent call last): 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task 
    uuid, retval, task_request, publish_result, 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done 
    self.store_result(task_id, result, state, request=request) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result 
    request=request, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result 
    self.set(self.get_key_for_task(task_id), self.encode(meta)) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set 
    return self.ensure(self._set, (key, value), **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure 
    **retry_policy) 
  File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time 
    return fun(*args, **kwargs) 
  File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set 
    pipe.execute() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute 
    return execute(conn, stack, raise_on_error) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction 
    connection.send_packed_command(all_cmds) 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command 
    self.connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect 
    self.on_connect() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect 
    if nativestr(self.read_response()) != 'OK': 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response 
    response = self._parser.read_response() 
  File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response 
    raise error 
ConnectionError: max number of clients reached 
  exc, exc_info.traceback))) 

My biggest problem with this is that this error is raised somewhere outside of my Try/Catch bloc. Hence, when this exception occurs, all my servers remain in the "Launching" mode rather than "Error".

How can I catch this exception and modify Server.status?

Answer 1

In redis 2.4 there is a hard coded limit of max number of connections which is 10,000. In redis 2.6+ you can specify the max number of clients in redis.conf. also this is not a problem of celery your broker redis refused to accept connections that's the problem.

Set the max number of clients that can be handled by redis simultaneously using redis CLI. Check out redis clients