Run dask scheduler and workers in Amazon's ECS (Fargate)

Question

I tried to run a scheduler and workers docker containers on Amazon's ECS. I'm using this example: https://docs.dask.org/en/latest/setup/docker.html

The scheduler works perfectly, I successfully connected to it from my local machine:

distributed.scheduler - INFO - Remove client Client-0ae5b0fa 
distributed.scheduler - INFO - Close client connection: Client-0ae5b0fa 
distributed.scheduler - INFO - Remove client Client-0ae5b0fa 
distributed.core - INFO - Starting established connection 
distributed.scheduler - INFO - Receive client connection: Client-0ae5b0fa

I try to run the worker the same way, with this command:

dask-worker tcp://SCHEDULER_PUBLIC_IP:8786

The worker is writing these logs and exits:

+ exec 'dask-worker tcp://SCHEDULER_PUBLIC_IP:8786'
/usr/bin/prepare.sh: line 30: /dask-worker tcp://SCHEDULER_PUBLIC_IP:8786: No such file or directory
+ '[' '' ']'
no environment.yml
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
+ '[' '' ']'
+ '[' '' ']'

I expected the worker to connect to the scheduler, because the same commands worked when I tried them on an EC2 instance. Also, I tried doing this with all ports open to tcp connections and still nothing.

Environment:

• Dask docker container version: 6bfa3b19b4be (1 AUG 2021) (latest)
• Fargate version: 1.4.0 (latest)
• Container has 2 vCPUs, 4 Gb memory

Answer 1

The problem was that my command was not comma delimited. it was:

dask-worker 1.1.1.1:8786

it is supposed to be:

dask-worker,1.1.1.1:8786

in order for docker to understand these are different arguments:

Command ["dask-worker","1.1.1.1:8786"]