访问 Sagemaker 管道中 Lambda 步骤的返回值

Question

I've added a Lambda Step as the first step in my Sagemaker Pipeline. It processes some data and creates 2 files as part of the output like so:

from sagemaker.workflow.lambda_step import LambdaStep, Lambda, LambdaOutput, LambdaOutputTypeEnum

# lamb_preprocess = LambdaStep(func_arn="")

output_param_1 = LambdaOutput(output_name="status", output_type=LambdaOutputTypeEnum.Integer)
output_param_2 = LambdaOutput(output_name="file_name_a_c_drop", output_type=LambdaOutputTypeEnum.String)
output_param_3 = LambdaOutput(output_name="file_name_q_c_drop", output_type=LambdaOutputTypeEnum.String)

step_lambda = LambdaStep(
    name="ProcessingLambda",
    lambda_func=Lambda(
        function_arn="arn:aws:lambda:us-east-1:xxxxxxxx:function:xxxxx"
    ),
    inputs={
        "input_data": input_data,
        "input_file": trigger_file,
        "input_bucket": trigger_bucket
    },
    outputs = [
        output_param_1, output_param_2, output_param_3
    ]
)

In my next step, I want to trigger a Processing Job for which I need to pass in the above Lambda function's outputs as it's inputs. I'm trying to do it like so:

inputs = [
    ProcessingInput(source=step_lambda.properties.Outputs["file_name_q_c_drop"], destination="/opt/ml/processing/input"),
    ProcessingInput(source=step_lambda.properties.Outputs["file_name_a_c_drop"], destination="/opt/ml/processing/input"),
]

However, when the processing step is trying to get created, I get a validation message saying

Object of type Properties is not JSON serializable

I followed the data dependency docs here: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_model_building_pipeline.html#lambdastep and tried accessing step_lambda.OutputParameters["file_name_a_c_drop"] too but it errored out saying 'LambdaStep' object has no attribute 'OutputParameters'

How do I properly access the return value of a LambdaStep in a Sagemaker pipeline?

Answer 1

You can access the output as follows - step_lambda.OutputParameters["output1"] . You don't need to add .properties

Answer 2

To access a LambdaStep output in another step you can do this:

step_lambda.properties.Outputs["file_name_a_c_drop"]

Answer 3

Try this

steplambda.properties.ProcessingOutputConfig.Outputs["file_name_q_c_drop"].S3Output.S3Uri