[英]How to parse stepfunction executionId to SageMaker batch transform job name?
I have created a stepfunction, the definition for this statemachine below ( step-function.json
) is used in terraform (using the syntax in this page: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTransformJob.html )我创建了一个 stepfunction,下面这个状态机的定义 ( step-function.json
) 在 terraform 中使用(使用此页面中的语法: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTransformJob .html )
The first time if I execute this statemachine, it will create a SageMaker batch transform job named example-jobname
, but I need to exeucute this statemachine everyday, then it will give me error "error": "SageMaker.ResourceInUseException", "cause": "Job name must be unique within an AWS account and region, and a job with this name already exists
.如果我第一次执行这个状态机,它会创建一个名为example-jobname
的 SageMaker 批量转换作业,但我需要每天执行这个状态机,然后它会给我错误"error": "SageMaker.ResourceInUseException", "cause": "Job name must be unique within an AWS account and region, and a job with this name already exists
。
The cause is because the job name is hard-coded as example-jobname
so if the state machine gets executed after the first time, since the job name needs to be unique, the task will fail, just wondering how I can add a string (something like ExecutionId at the end of the job name).原因是因为作业名称被硬编码为example-jobname
所以如果 state 机器在第一次执行后,由于作业名称需要唯一,任务将失败,只是想知道如何添加一个字符串(类似于作业名称末尾的 ExecutionId 之类的东西)。 Here's what I have tried:这是我尝试过的:
I added "executionId.$": "States.Format('somestring {}', $$.Execution.Id)"
in the Parameters
section in the json file, but when I execute the task I got error "error": "States.Runtime", "cause": "An error occurred while executing the state 'SageMaker CreateTransformJob' (entered at the event id #2). The Parameters '{\"BatchStrategy\":\"SingleRecord\",..............\"executionId\":\"somestring arn:aws:states:us-east-1:xxxxx:execution:xxxxx-state-machine:xxxxxxxx72950\"}' could not be used to start the Task: [The field \"executionId\" is not supported by Step Functions]"}
我在 json 文件的Parameters
部分添加了"executionId.$": "States.Format('somestring {}', $$.Execution.Id)"
,但是当我执行任务时出现错误"error": "States.Runtime", "cause": "An error occurred while executing the state 'SageMaker CreateTransformJob' (entered at the event id #2). The Parameters '{\"BatchStrategy\":\"SingleRecord\",..............\"executionId\":\"somestring arn:aws:states:us-east-1:xxxxx:execution:xxxxx-state-machine:xxxxxxxx72950\"}' could not be used to start the Task: [The field \"executionId\" is not supported by Step Functions]"}
I modified the jobname in the json file to "TransformJobName": "example-jobname-States.Format('somestring {}', $$.Execution.Id)",
, when I execute the statemachine, it gave me error: "error": "SageMaker.AmazonSageMakerException", "cause": "2 validation errors detected: Value 'example-jobname-States.Format('somestring {}', $$.Execution.Id)' at 'transformJobName' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}; Value 'example-jobname-States.Format('somestring {}', $$.Execution.Id)' at 'transformJobName' failed to satisfy constraint: Member must have length less than or equal to 63
我将json文件中的jobname修改为"TransformJobName": "example-jobname-States.Format('somestring {}', $$.Execution.Id)",
时报错: "error": "SageMaker.AmazonSageMakerException", "cause": "2 validation errors detected: Value 'example-jobname-States.Format('somestring {}', $$.Execution.Id)' at 'transformJobName' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}; Value 'example-jobname-States.Format('somestring {}', $$.Execution.Id)' at 'transformJobName' failed to satisfy constraint: Member must have length less than or equal to 63
I really run out of ideas, can someone help please?我真的没有主意了,有人可以帮忙吗? Many thanks.非常感谢。
So as per the documentation , we should be passing the parameters in the following format所以根据文档,我们应该以下列格式传递参数
"Parameters": {
"ModelName.$": "$$.Execution.Name",
....
},
If you take a close look this is something missing from your definition, So your step function definition should be something like below:如果您仔细查看您的定义中缺少的内容,那么您的步骤 function 定义应该如下所示:
either任何一个
"TransformJobName.$": "$$.Execution.Id",
OR要么
"TransformJobName.$: "States.Format('mytransformjob{}', $$.Execution.Id)"
full State machine definition:完整的 State 机器定义:
{
"Comment": "Defines the statemachine.",
"StartAt": "Generate Random String",
"States": {
"Generate Random String": {
"Type": "Task",
"Resource": "arn:aws:lambda:eu-central-1:1234567890:function:randomstring",
"ResultPath": "$.executionid",
"Parameters": {
"executionId.$": "$$.Execution.Id"
},
"Next": "SageMaker CreateTransformJob"
},
"SageMaker CreateTransformJob": {
"Type": "Task",
"Resource": "arn:aws:states:::sagemaker:createTransformJob.sync",
"Parameters": {
"BatchStrategy": "SingleRecord",
"DataProcessing": {
"InputFilter": "$",
"JoinSource": "Input",
"OutputFilter": "xxx"
},
"Environment": {
"SAGEMAKER_MODEL_SERVER_TIMEOUT": "300"
},
"MaxConcurrentTransforms": 100,
"MaxPayloadInMB": 1,
"ModelName": "${model_name}",
"TransformInput": {
"DataSource": {
"S3DataSource": {
"S3DataType": "S3Prefix",
"S3Uri": "${s3_input_path}"
}
},
"ContentType": "application/jsonlines",
"CompressionType": "Gzip",
"SplitType": "Line"
},
"TransformJobName.$": "$.executionid",
"TransformOutput": {
"S3OutputPath": "${s3_output_path}",
"Accept": "application/jsonlines",
"AssembleWith": "Line"
},
"TransformResources": {
"InstanceType": "xxx",
"InstanceCount": 1
}
},
"End": true
}
}
}
In the above definition the lambda could be a function which parses the execution id arn which I am passing via the parameters section:在上面的定义中,lambda 可能是一个 function,它解析我通过参数部分传递的执行 ID arn:
def lambda_handler(event, context):
return(event.get('executionId').split(':')[-1])
Or if you dont wanna pass the execution id, it can simply return the random string like或者,如果您不想传递执行 ID,它可以简单地返回随机字符串,例如
import string
def lambda_handler(event, context):
return(string.ascii_uppercase + string.digits)
you can generate all kinds of random string or do generate anything in the lambda and pass that to the transform job name.您可以生成各种随机字符串或在 lambda 中生成任何内容并将其传递给转换作业名称。
I would like to throw another idea.我想提出另一个想法。 You can use, if applicable, also another executionId or other unique identifier from the previous task.如果适用,您还可以使用上一个任务中的另一个 executionId 或其他唯一标识符。
I am triggering the BatchTransform job after a successful GlueJob.我在 GlueJob 成功后触发 BatchTransform 作业。 Therefore, I can take the output variables and concatenate in the BatchTransform job to be used a new TransformJobName.因此,我可以使用 output 变量并在 BatchTransform 作业中串联以使用新的 TransformJobName。
"TransformJobName.$": "States.Format('scoring-titanic-{}', $.CompletedOn)"
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.