Problem starting trivial standalone spark app: java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/StaticSQLConf$

Question

I run spark tests several to many times daily but have run into a new problem when launching a simple standalone app. I have tried on both spark 2.3.0 and 2.4.0 with the same results.

  def main(args: Array[String]): Unit = {
    val spark =   SparkSession.builder.appName("bestRoutes").master("local").getOrCreate
    // do stuff with the session..
  }

This results in the error / stacktrace shown here:

19/03/09 11:02:24 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.4:54406 with 2004.6 MB RAM, BlockManagerId(driver, 192.168.0.4, 54406, None)
19/03/09 11:02:24 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.4, 54406, None)
19/03/09 11:02:24 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.4, 54406, None)
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/StaticSQLConf$
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
    at task1.AirData$$anonfun$2.apply(AirData.scala:52)
    at task1.AirData$$anonfun$2.apply(AirData.scala:52)
    at scala.Option.getOrElse(Option.scala:121)
    at task1.AirData$.main(AirData.scala:51)
    at task1.AirData.main(AirData.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.internal.StaticSQLConf$
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 6 more
19/03/09 11:02:24 INFO SparkContext: Invoking stop() from shutdown hook
19/03/09 11:02:24 INFO SparkUI: Stopped Spark web UI at http://192.168.0.4:4040

I am using a tried and true maven pom.xml - and the relevant spark portions are shown here:

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>

And then the required shading - which is standard part of spark projects

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <createDependencyReducedPom>false</createDependencyReducedPom>
                        <relocations>
                            <relocation>
                                <pattern>com.google.common</pattern>
                                <shadedPattern>shaded.com.google.common</shadedPattern>
                            </relocation>
                            <relocation>
                                <pattern>org.glassfish.hk2</pattern>
                            </relocation>

                        </relocations>
                        <transformers>
                            <transformer
                                    implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <manifestEntries>
                                    <Main-Class>a2a.TimeSeries</Main-Class>
                                    <Build-Number>1.0</Build-Number>
                                </manifestEntries>
                            </transformer>
                        </transformers>
                        <artifactSet>
                            <excludes>
                                <exclude>org.apache.spark:*</exclude>
                                <exclude>org.apache.hadoop:*</exclude>
                            </excludes>
                        </artifactSet>
                        <filters>
                            <filter>
                                <artifact>*.*</artifact>
                                <excludes>
                                    <!--<exclude>org.apache.maven:lib:tests</exclude>-->
                                    <exclude>log4j:log4j:jar:</exclude>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                    <exclude>META-INF/ECLIPSE*</exclude>
                                    <exclude>META-INF/license/*</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <minimizeJar>false</minimizeJar>
                    </configuration>
                </execution>
            </executions>
        </plugin>

Answer 1

StaticSQLConf class is part of artifact spark-catalyst_${scala.binary.version} which is used as provided maven scope. Hence spark runtime can't find it. It should be compile.

   <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>compile</scope>
    </dependency>