Celery Flower with Multiple Workers in Different Docker Containers

Question

I've been up and down StackOverflow and Google, but I can't seem to come close to an answer.

tl;dr How do I register a dockerized Celery worker in a dockerized Flower dashboard? How do I point the worker to the Flower dashboard so that the dashboard "knows" about it?

I have 2 FastAPI apps, both deployed with docker-compose.yml files. The first app's compose file looks like this:

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: ./compose/local/fastapi/Dockerfile
    image: app_web
    # '/start' is the shell script used to run the service
    command: /start
    volumes:
      - .:/app
    ports:
      - 8010:8000
    env_file:
      - .env/.dev-sample
    depends_on:
      - redis

  redis:
    image: redis:6-alpine

  celery_worker:
    build:
      context: .
      dockerfile: ./compose/local/fastapi/Dockerfile
    image: app_celery_worker
    command: /start-celeryworker
    volumes:
      - .:/app
    env_file:
      - .env/.dev-sample
    depends_on:
      - redis

  flower:
    build:
      context: .
      dockerfile: ./compose/local/fastapi/Dockerfile
    image: app_celery_flower
    command: /start-flower
    volumes:
      - .:/app
    env_file:
      - .env/.dev-sample
    ports:
      - 5557:5555
    depends_on:
      - redis

So this app is responsible for creating the Celery Flower dashboard.

The second app's compose file looks like:

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: ./compose/local/fastapi/Dockerfile
    image: app_two_web
    # '/start' is the shell script used to run the service
    command: /start
    volumes:
      - .:/app
    ports:
      - 8011:8000
    env_file:
      - .env/.dev-sample
    depends_on:
      - redis

  redis:
    image: redis:6-alpine

  celery_worker:
    build:
      context: .
      dockerfile: ./compose/local/fastapi/Dockerfile
    image: app_two_celery_worker
    command: /start-celeryworker
    volumes:
      - .:/app
    env_file:
      - .env/.dev-sample
    depends_on:
      - redis

I can't get this second app's worker to register in the Celery Flower dashboard running on port 5557. Everything works fine, and I can even launch a second Flower dashboard with the second app if on a different port, but I can't seem to connect the second worker to the first app's Flower dashboard.

This is what main.py looks like, for both apps.

from project import create_app

app = create_app()
celery = app.celery_app


def celery_worker():
    from watchgod import run_process
    import subprocess

    def run_worker():
        subprocess.call(
            ["celery", "-A", "main.celery", "worker", "-l", "info"]
        )

    run_process("./project", run_worker)


if __name__ == "__main__":
    celery_worker()

Thanks for any ideas that I can throw at this.

Answer 1

• First enable event monitoring by putting "-E" in your worker container "command:"
• Second, specify environment variable C_FORCE_ROOT in every worker services in your docker-compose configuration.