How to run one-off Docker containers locally (triggered from within a container)

Question

When hosting applications on Heroku I often trigger one-off dynos via the Heroku API from within the code to do heavy lifting in the background. I recently set up some stacks on AWS and followed a similar pattern by using AWS ECS run task.

I am not using long running queue workers for this as hardware resources vary heavily according to the specific task and usually the workload occurs in peaks.

For local development, I usually skipped this topic by either executing the background tasks within the running container or triggering the background command manually from the console. What would be a good approach for running one-off containers locally?

Answer 1

ECS supports scheduled tasks, if you know when your peaks are planned for you can use scheduled tasks to launch fargate containers on a schedule.

If you don't, what we did was write a small API Gateway -> Lambda function that basically dynamically launches fargate containers with a few variables defined in the POST to the API Gateway endpoint like CPU/Mem/port etc...Or pre-create task definitions and just pass the task def to the api, which is another option if you know what the majority of your "settings" should be for the container.

Answer 2

You can simply call ECS RunTask API call from inside the container.

All you need is to setup ECS Task role to have runtask permissions and to have either aws cli or any aws sdk in container to call runtask call.

Answer 3

you can pass docker socket as a volume

volumes:
     - /var/run/docker.sock:/var/run/docker.sock

After it you can run docker commands inside the container and they will be executed by docker on the host machine.

In particular you can run

docker run ...

or

docker start ...

(may be you will have to install docker in your container via commands in Dockerfile )