[英]AWS Batch - How to access AWS Batch environment variables within python script running inside Docker container
I have a Docker container which executes a python script inside it as the ENTRYPOINT. 我有一个Docker容器,它在其中执行一个python脚本作为ENTRYPOINT。 This is the DockerFile 这是DockerFile
FROM python:3
ADD script.py /
EXPOSE 80
RUN pip install boto3
RUN pip install uuid
ENTRYPOINT ["python","./script.py"]
This is the Python script: 这是Python脚本:
import boto3
import time
import uuid
import os
guid = uuid.uuid4()
timestr = time.strftime("%Y%m%d-%H%M%S")
job_index = os.environ['AWS_BATCH_JOB_ARRAY_INDEX']
filename = 'latest_test_' + str(guid) + '_.txt'
with open(filename, 'a+') as f:
data = job_index
f.write(data)
client = boto3.client(
's3',
# Hard coded strings as credentials, not recommended.
aws_access_key_id='',
aws_secret_access_key=''
)
response = client.upload_file(filename, 'api-dev-dpstorage-s3', 'docker_data' + filename + '.txt')
with open('response2.txt', 'a+') as f:
f.write('all done')
exit
It is simply designed to create a file, write the job array index into the file and push it to an S3 Bucket. 它只是设计用于创建文件,将作业数组索引写入文件并将其推送到S3 Bucket。 The job array index from AWS Batch is being sourced from one of the pre-defined environment variables. AWS Batch的作业数组索引来自其中一个预定义的环境变量。 I have uploaded the image to AWS ECR, and have set up an AWS Batch to run a job with an array of 10. This should execute the job 10 times, with my expectation that 10 files are dumped into S3, each containing the array index of the job itself. 我已将图像上传到AWS ECR,并已设置AWS Batch以运行数组为10的作业。这应该执行作业10次,我期望将10个文件转储到S3中,每个文件都包含数组索引工作本身。
If I don't include the environment variable and instead just hard code a value into the text file, the AWS Batch job works. 如果我不包含环境变量,而只是将值硬编码到文本文件中,则AWS Batch作业可以正常工作。 If I include the call to os.environ to get the variable, the job fails with this AWS Batch error: 如果我包含对os.environ的调用以获取变量,则作业将因此AWS Batch错误而失败:
Status reasonEssential container in task exited
I'm assuming there is an issue with how I'm trying to obtain the environment variable. 我假设我如何尝试获取环境变量存在问题。 Does anyone know how I could correctly reference either one of the built in environment variables and/or a custom environment variable defined in the job? 有谁知道如何正确引用内置环境变量和/或作业中定义的自定义环境变量?
AWS provides docker env
configuration by job definition parameters , where you specify: AWS通过作业定义参数提供docker env
配置,您可以在其中指定:
"environment" : [
{ "AWS_BATCH_JOB_ARRAY_INDEX" : "string"},
]
This will be turned into docker env parameter: 这将变成docker env参数:
$ docker run --env AWS_BATCH_JOB_ARRAY_INDEX=string $container $cmd
Thus can be accessed by: 因此可以访问 :
import os
job_id = os.environ['AWS_BATCH_JOB_ARRAY_INDEX']
But watch out if you are passing sensitive data in this way, it is not wise to pass credentials in plain text. 但请注意,如果以这种方式传递敏感数据,以明文形式传递凭据是不明智的。 Instead, in this case, you may want to create a compute environment . 相反,在这种情况下,您可能想要创建计算环境 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.