Could not parse Master URL: 'yarn' on AWS EMR

Question

I am looking to run a spark application as a step on a cluster with yarn as the master. While creating the cluster, I used the following configuration:

I get the following error while running my application:

Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL: 'yarn'

I also tried adding the spark-yarn 3.0.1 as a build dependency when compiling with sbt assembly, but I still get the same error. I would really appreciate if someone can help me find the right track to get rid of this error. Thanks!

Answer 1

Can you please add spark yarn dependency in your build path and try again ?

Since you are using spark 3.0.1 and SBT, you can use the below dependency.

libraryDependencies += "org.apache.spark" %% "spark-yarn" % "3.0.1" % "provided"