Celery + Flask + Docker, consumer: Cannot connect to amqp://admin:**@rabbit:5672/myhost: failed to resolve broker hostname

Question

Background

I am building a web application that uses Flask for the backend framework. The application uses Celery to handle all the time-consuming tasks as background tasks as to not block the backend thread. I use RabbitMQ as the message broker for Celery workers. I bundled each service using docker-compose.

Problem

The app has been working well until the past few days, and all of sudden, Celery workers keep failing to connect to the message broker with the error message [ERROR/MainProcess] consumer: Cannot connect to amqp://admin:**@rabbit:5672/myhost: failed to resolve broker hostname.

Directory structure and code

I put together files and directories for minimally reproducible example.

debug/
├── code
│   ├── dev.Dockerfile
│   ├── my_app
│   │   ├── celery_app.py
│   │   ├── config.py
│   │   ├── extensions.py
│   │   ├── __init__.py
│   │   ├── my_tasks.py
│   │   └── test_app.py
│   └── requirements.txt
└── docker-compose_dev.yml

docker-compose_dev.yml

version: "3.7"
services:
  rabbit:
    image: rabbitmq:3.8.5-management
    ports:
      - '15673:15672' # in case user has rabbitMQ installed on host
    expose:
      - "5672"
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
      - RABBITMQ_DEFAULT_VHOST=myhost

  non_working_worker:
    build:
      context: ./code
      dockerfile: dev.Dockerfile
    command: "celery worker -A my_app.celery_app:app -l info"
    volumes:
      - ./code:/code
    links:
      - rabbit

  working_worker:
    build:
      context: ./code
      dockerfile: dev.Dockerfile
    command: "celery worker -A my_app.my_tasks:app -l info"
    volumes:
      - ./code:/code
    links:
      - rabbit

dev.Dockerfile

FROM continuumio/miniconda3

# Make /backend working directory; flask code lives here
WORKDIR /code

# Install from requirements.txt using pip
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
RUN rm requirements.txt

requirements.txt

luigi==2.8.11
plotnine==0.7.0
celery==4.4.6
flask==1.1.2
flask-cors
flask-socketio
Flask-Mail
eventlet

test_app.py

import eventlet
eventlet.monkey_patch()

from flask import Flask
from my_app.extensions import celery

def create_app():
    """
    Application factory. Create application here.
    """
    app = Flask(__name__)
    app.config.from_object("my_app.config")

    return app

def init_celery(app=None):
    """
    Initialize Celery App
    """
    app = app or create_app()
    app.config.from_object("my_app.config")

    # Set celery worker configuration
    # Use this to load config information from flask config file
    celery.conf.broker_url = app.config["CELERY_BROKER_URL"]
    celery.conf.result_backend = app.config["CELERY_RESULT_BACKEND"]

    class ContextTask(celery.Task):
        """Make celery tasks work with Flask app context"""

        def __call__(self, *args, **kwargs):
            with app.app_context():
                return self.run(*args, **kwargs)

    celery.Task = ContextTask

    return celery

config.py

# RabbitMQ
CELERY_BROKER_URL='pyamqp://admin:mypass@rabbit/myhost'
CELERY_RESULT_BACKEND='rpc://'

extensions.py

from celery import Celery

celery = Celery()

celery_app.py

from my_app.test_app import init_celery

app = init_celery()

my_tasks.py

from celery import Celery


app = Celery()

app.conf.broker_url = 'pyamqp://admin:mypass@rabbit/myhost'
app.conf.result_backend = 'rpc://'

What I've tried

Followings are the things I've tried, but didn't work.

1. RabbitMQ isn't launching properly?
   • a. It launches properly with given username, password, and vhost. (can check using the management plugin @ localhost:15673)
2. RabbitMQ launches after the Celery workers start, so the workers can't find the broker?
   • a. Celery has retry feature, so it will keep on retrying until message broker is up running.
3. Network issue?
   • a. I've tried with/without links to specify service name alias, but still didn't work.
   • b. Note I've already specified broker name as rabbit as specified in the config.py file instead of localhost
   • c. I've tried using both the default network docker-compose creates, and custom network, but both failed.
4. Interestingly, Celery app instance in my_tasks.py works (it's named as working_worker in the docker-compose file), but Celery app instance in the Flask factory pattern does not work (it'a named as non_working_worker in the compose file)
   • a. Again, it shows that RabbitMQ is working fine, but something funky is going on with the Flask factory pattern style Celery app instantiation.

I spent past few days trying to fix this issue and searching for similar problems on internet, but had no luck doing so.

I know it's a fairly long post, but any help/suggestions would greatly be appreciated.

docker-compose version

docker-compose version 1.25.3, build d4d1b42b
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

docker version

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:45:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:44:07 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Answer 1

I had a similar issue that I was able to resolve by specifying the version of dnspython, one of eventlets dependencies, to 1.16.0 in my requirements.txt above eventlet. It looks like eventlet is not compatible with the latest version of dnspython, more info here https://github.com/eventlet/eventlet/issues/619