如何使用 docker-compose 构建不同配置的 celery 工作人员？

Question

I am using docker-compose to build and run celery workers.

services:
    ...

    worker-a:
        command: celery -A tasks worker -l "DEBUG" --concurrency=1 -Q qname_a
        restart: always
        environment:
            WORKER_TYPE: a_setting
            ...
        depends_on:
            - ...

    worker-b:
        command: celery -A tasks worker -l "DEBUG" --concurrency=1 -Q qname_b
        restart: always
        environment:
            WORKER_TYPE: b_setting
            ...
        depends_on:
            - ...

How can I build 2 different worker images (worker-a and worker-b), which perform the same task (same code), but use different configurations (based on WORKER_TYPE) and listen to different queues?

Answer 1

If you are aiming to avoid having to repeat an almost identical piece of code, YAML anchors are great for defining reusable blocks.

For example, in the below configuration all the configuration for worker-a can be used by any other service by calling <<: *default , and then overriding any value as needed:

version: '3'

services:
  worker-a: &default
    image: alpine
    command: env
    environment:
      WORKER_TYPE: a_setting
    # ... more definitions

  worker-b:
    <<: *default
    environment:
      WORKER_TYPE: b_setting

Notice that if you want nested elements - such as environment - to also have common values, you may need to do something like this:

version: '3'

services:
  worker-a: &default
    image: alpine
    command: env
    environment: &environment
      COMMON_VAR_FOR_ALL: value
      WORKER_TYPE: a_setting
    # ... more definitions

  worker-b:
    <<: *default
    environment:
      <<: *environment
      WORKER_TYPE: b_setting

There is a good writeup on the subject here and here .