AWS Step Functions EMR PySpark Task Step failed

Question

I have a working EMR step that takes around 500 seconds. The working EMR controller log shows:

2021-09-19T18:36:59.786Z INFO StepRunner: Created Runner for step 7
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --deploy-mode cluster s3://emr/scripts/aggregate.py --day 20210915 --base_uri s3://'
...
2021-09-19T18:46:35.934Z INFO Step succeeded with exitCode 0 and took 576 seconds

When I try to run the same step using Step Functions the spark-submit code looks identical but there is an error:

2021-09-19T18:36:59.786Z INFO StepRunner: Created Runner for step 7
INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --deploy-mode cluster s3://emr/scripts/aggregate.py --day 20210915 --base_uri s3://'
...
2021-09-22T09:56:07.309Z WARN Step failed with exitCode 1 and took 2 seconds

STDERR shows: Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "spark-submit" (in directory "."): error=2, No such file or directory

What is the correct way to run a pyspark script using step functions?

Step Definition:

"EMR AddStep - Aggregate Daily": {
  "Type": "Task",
  "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
  "Parameters": {
    "ClusterId.$": "$.cluster.Cluster.Id",
    "Step": {
      "Name": "Aggregate Daily",
      "ActionOnFailure": "CONTINUE",
      "HadoopJarStep": {
        "Jar": "command-runner.jar",
        "Args": [
          "spark-submit",
          "--deploy-mode",
          "cluster",
          "s3://emr/scripts/aggregate.py",
          "--day",
          "20210915",
          "--base_uri",
          "s3://"
        ]
      }
    }
  }

Answer 1

The issue was that the EMR cluster was created without Spark option.

Solution was to add "Name": "Spark" under "Applications" property in the step functions create cluster task:

    "EMR CreateCluster": {
      "Type": "Task",
      "Resource": "arn:aws:states:::elasticmapreduce:createCluster.sync",
      "Parameters": {
        "Name": "ExampleCluster",
        "VisibleToAllUsers": true,
        "ReleaseLabel": "emr-5.33.0",
        "Applications": [
          {
            "Name": "Hive",
            "Name": "Spark"
          }
        ],