使用 EMR 无服务器的 AWS Step 函数编排作业

Question

Recently Amazon launched EMR Serverless and I want to repurpose my exiting data pipeline orchestration that uses AWS Step Functions : There are steps that create EMR cluster, run some lambda functions, submit Spark Jobs (mostly Scala jobs using spark-submit) and finally terminate the cluster. All these steps are of sync type ( arn:aws:states:::elasticmapreduce:addStep.sync )

There are documentation and github samples that describe submitting jobs from orchestration framework such as AirFlow but there is nothing that describes how to use AWS Step Function with EMR Serverless. Any help in this regard is appreciated.

Primarily I am interested in repurposing task step function of type arn:aws:states:::elasticmapreduce:addStep.sync that takes parameters such as ClusterId but in case of EMR Serverless there is no such id.

In summary is there equivalent of Call Amazon EMR with Step Functions for EMR Serverless?

Answer 1

Currently there is no direct integration of EMR Serverless with Step Functions. However a possible solution is adding a Lambda Layer on top and use the SDK to create emr serverless applications and submit jobs. However you would need an additional lambda to implement a poller that tracks the success of the jobs (in case of interdependent jobs) as it is highly likely that the emr job will outrun the 15 min runtime limitation of the lambda.

Answer 2

As @KhalidJahangeer said, there's still no direct integration between EMR serverless and Step Functions, but you might want to check the new capability AWS SDK Service Integrations .

This option allows you to make use of services' APIs that aren't directly integrated with Step Functions, previously if the service integration was not available, you had to code the integration in an AWS Lambda function but now this last step can be removed, as AWS explains:

You can create state machines that use AWS SDK Service Integrations with Amazon States Language (ASL), AWS Cloud Development Kit (AWS CDK), or visually using AWS Step Function Workflow Studio. To get started, create a new Task state. Then call AWS SDK services directly from the ASL in the resource field of a task state. To do this, use the following syntax.

 arn:aws:states:::aws-sdk:serviceName:apiAction.[serviceIntegrationPattern]

Some important things to keep in mind are:

• Call AWS SDK services directly from the ASL in the resource field of a task state. To do this, use the following syntax: arn:aws:states:::aws-sdk:serviceName:apiAction.[serviceIntegrationPattern]
• Use camelCase for apiAction names in the Resource field, such as copyObject , and use PascalCase for parameter names in the Parameters field, such as CopySource .
• Step Functions can't autogenerate IAM policies for most AWS SDK Service - - Integrations, so you need to add those to the IAM role of the state machine manually.
• Take advantage of ASL intrinsic functions, as those allow you to manipulate the data and avoid using Lambda functions for simple transformations.