Artifacts (.jar) Generated using with Intellij Idea Scala Spark throws ClassNotFoundException with spark-submit

Question

I have gone through various related suggestions, but I am still struggling.

What I have:

Spark:2.3.1
Scala:2.11.8
OS: Windows 10
IDE:Intellij Idea
Code:
     package: testpackage
                      merge.scala -->has do() method
                      mymain.scala -->has main() method and inside that merge.do()

Project Settings--Artifacts-->
                              Main Class-->testpackage.mymain
                              Class Path--><blank>

What is working:

1. Running properly on IDE
2. JAR creation : Artifacts are getting generated properly as testpackage.jar
3. I can see the classes(along with various other libraries) when I open testpackage.jar in winrar as:
   testpackage\merge$$anonfun$1.class
   testpackage\merge$$anonfun$2.class
   testpackage\merge$.class
   testpackage\merge.class
   testpackage\mymain$.class
   testpackage\mymain.class

What is not working

spark-submit from command prompt throws exception:

java.lang.ClassNotFoundException: testpackage.mymain
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

More info:

1. spark-submit executed from jar folder (~out\artifacts\testpackage_jar)
2. the testpackage.jar is about 128MB

Same exception if I try:

spark-submit testpackage.jar C:\temp\input.csv
spark-submit --class testpackage.mymain testpackage.jar C:\temp\input.csv
spark-submit --class mymain testpackage.jar C:\temp\input.csv

Also tried with following statement in build.sbt

mainClass in (Compile, packageBin) := Some("testpackage.mymain")

Also tried with following statement in build.sbt

mainClass in (Compile, packageBin) := Some("mymain")

Also tried by keeping jar in Spark bin folder, in my machine it is

C:\Spark\spark-2.3.1-bin-hadoop2.7\bin

Tried with --mater local[*], and a few other combinations.

Appreciate your help!!

Answer 1

Finally I found the solution. Just in case anyone else bump on this. Make sure that you have only 1 entry, which is "'your package' compile output" under "'your package.jar'" on Project Structure-->Artifacts-->Output Layout

This is strange, still I don't know why it will not work!

Now my jar size is 10KB & works good! :)