Redis loosing connection when used with Celery groups or chain - throwing ERROR/MainProcess] Connection to Redis lost: Retry (0/20) now

Question

My problem is with redis showing the following error in terminal when i start my worker using celery task.

I am lucky to have reproduce the error using a modified version of chord in celery documentation . It seems the problem happens with chord or where I have too many groups running in parallel ie chord(add.s(i, i) for i in range(1, num))(list_add.s()) or group(add.s(i, i) for i in range(1, num))()

Below is my code sample

@task
def add(x, y):
    return [x, y, x + y, "Next"]

@task
def list_add(nums):
    numbers = []
    count = 1
    for i in nums:
        print("{}). {}".format(count, i))
        numbers.extend(i)
        count += 1
    print(numbers)
    return numbers

@task
def foo(num):
    return chord(add.s(i, i) for i in range(1, num))(list_add.s())

Below is part of my terminal output with traceback.

    [2015-11-04 20:36:14,912: INFO/MainProcess] Received task: b2b.tasks.add[b87fdc44-e759-4224-bce4-11f9468d12b3]
[2015-11-04 20:36:14,913: INFO/MainProcess] Received task: b2b.tasks.add[120f5bf2-b962-4424-894b-d6f0ca56102b]
[2015-11-04 20:36:14,914: INFO/MainProcess] Task b2b.tasks.bar[9dc93c75-6404-4db3-a685-ff91460e1adb] succeeded in 1.00830382s: <AsyncResult: c891df3e-aa5c-4c9f-ad3f-30abe3b3ccc1>

    [2015-11-04 19:44:34,922: ERROR/MainProcess] Connection to Redis lost: Retry (0/20) now.
    [2015-11-04 19:44:34,922: ERROR/MainProcess] Connection to Redis lost: Retry (1/20) in 1.00 second.
    ...
    /Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/celery/app/trace.py:365: RuntimeWarning: Exception raised outside body: ConnectionError('Error 8 connecting to localhost:6379. nodename nor servname provided, or not known.',):
    Traceback (most recent call last):
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/celery/app/trace.py", line 235, in trace_task
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/celery/backends/base.py", line 256, in store_result
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/celery/backends/base.py", line 490, in _store_result
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/celery/backends/redis.py", line 160, in set
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/celery/backends/redis.py", line 149, in ensure
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/kombu/utils/__init__.py", line 243, in retry_over_time
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/celery/backends/redis.py", line 169, in _set
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/redis/client.py", line 2593, in execute
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/redis/client.py", line 2447, in _execute_transaction
      File "/Users/Me/.virtualenvs/djangoscrape/lib/python2.7/site-packages/redis/connection.py", line 532, in send_packed_command
      File "/Users/Michael/.virtualenvs/djangoscrape/lib/python2.7/site-packages/redis/connection.py", line 436, in connect
    ConnectionError: Error 8 connecting to localhost:6379. nodename nor servname provided, or not known.

This is the command i use to start my worker in terminal celery -A scraper worker -P eventlet -c 1000 -l info

It works fine when foo() is passed 20 but with 1000 or greater the error shows up.

>>> a = foo.delay(20)     # works 
>>> a = foo.delay(1000)   # fails

Please, kindly suggest how this can be solved if you have an idea and thanks in advance.

Answer 1

I finally figured it out with help from IRC and Greg from redis user groups

By changing the limit of opened files on my Mac OSX Yosemite to 65536 the problem was solved.

I hope this helps someone someday.