在现有EMR上运行Scala Spark作业

Question

I have Spark Job aggregationfinal_2.11-0.1 jar which I am running on my machine.the composition of it is as follows :

package deploy
    object FinalJob {
      def main(args: Array[String]): Unit = {
        val spark = SparkSession
          .builder()
          .appName(s"${this.getClass.getSimpleName}")
          .config("spark.sql.shuffle.partitions", "4")
          .getOrCreate()

    //continued code
    }
    }

When I am running this code in local mode, it is running fine but when I am deploying this on the EMR cluster with putting its jar in main node.It is giving error as :

ClassNotFoundException : deploy.FinalJob

What am i missing here?

Answer 1

The best option is to deploy your uber jar(you can use sbt assembly plugin to build jar) to s3 and add spark step to EMR cluster. Please check: http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-submit-step.html

Answer 2

try to unjar it to some folder and look for target/classes using the below command jar -xvf myapp.jar. If the target classes is not containing the class you are executing then there is an issue with the way you build your jar. I would recommend maven assembly to be in your pom for packaging.