基于相同的任务定义但使用不同的环境变量运行多个ECS任务

Question

I have a task to process large amount of data, so data is batched into lots of parts. I have written the task definition for this type of work but now I know only how to set up them manually via registering multiple task definitions for each env.

sample where each task has its own env BATCH_ID

  aws ecs register-task-definition --cli-input-json file://taskdef1.json  
  aws ecs run-task --cluster $cluster --task-definition process_data_1

  aws ecs register-task-definition --cli-input-json file://taskdef2.json  
  aws ecs run-task --cluster $cluster --task-definition process_data_2

It would be even nice to have some .manifest file of all task arns put for cluster.

Is there someway to run multiple similar ECS tasks with different env params in more elegant way then creating enormous amount of different taskdefs files?

Thanks for any help and suggestions

Answer 1

You can override environmental variable when you run the task with --overrides flag. I use this all the time; you can either override an existing environmental variable (defined in task definition) or simply add a new one. --overrides accepts only JSON (no shorthand syntax); in your case, it would look like:

{
  "containerOverrides": [
    {
      "name": "containerNameFromTaskDefinition",
      "environment": [
        {
          "name": "BATCH_ID",
          "value": "sampleBatchId"
        }
      ]
    }
  ]
}

and the command:

aws ecs run-task --cluster $cluster --task-definition process_data_1 --overrides {"containerOverrides":[...]} 

You can use --overrides to override even more things of course: https://docs.aws.amazon.com/cli/latest/reference/ecs/run-task.html

Answer 2

I may be misunderstanding the problem here, but could you just put the variable in your cli input .json and use sed to swap out the batch ID before you run the task? I've done this for application deployments when I need to swap out env vars.

First, put the variable you want to replace in your taskdef template file (like %BATCH_ID%)

Then do something like this: (NOTE: Bash script is off the top of my head and may need fixing)

sed 's/%BATCH_ID%/$BATCH_ID' generic_taskdef.json > process_data_$BATCH_ID.json 
aws ecs register-task-definition --cli-input-json file://process_data_$BATCH_ID.json
aws ecs run-task --cluster $cluster --task-definition process_data_$BATCH_ID