Databricks - 火花提交错误 | org.springframework.core.ResolvableType.forInstance(Ljava/lang/Object;)Lorg/springframework/core/ResolvableType

Question

Spark-submit in Databricks cluster.. is giving this error. I am using

Spark 3.1.2

Scala 2.12

Springframeworkboot 2.6.3

However spark-submit is running good in mac and other local machines. But not in Databricks cluster.

Note: Also Added spring-core-5.3.15.jar in the maven build jar.

Not able to find the root cause for this error. Appreciate your inputs on this.

java.lang.NoSuchMethodError: org.springframework.core.ResolvableType.forInstance(Ljava/lang/Object;)Lorg/springframework/core/ResolvableType;
                at org.springframework.context.event.SimpleApplicationEventMulticaster.resolveDefaultEventType(SimpleApplicationEventMulticaster.java:149)
                at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131)
                at org.springframework.boot.context.event.EventPublishingRunListener.starting(EventPublishingRunListener.java:79)
                at org.springframework.boot.SpringApplicationRunListeners.lambda$starting$0(SpringApplicationRunListeners.java:56)
                at java.util.ArrayList.forEach(ArrayList.java:1259)
                at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
                at org.springframework.boot.SpringApplicationRunListeners.starting(SpringApplicationRunListeners.java:56)
                at org.springframework.boot.SpringApplication.run(SpringApplication.java:293)
                at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303)
                at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292)
                at com.ApplicationAgent.main(ApplicationExportAgent.java:9)
                at $linefd8c230552274eceb0e9701075fd142425.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command--1:1)
                at $linefd8c230552274eceb0e9701075fd142425.$read$$iw$$iw$$iw$$iw$$iw.<init>(command--1:43)
                at $linefd8c230552274eceb0e9701075fd142425.$read$$iw$$iw$$iw$$iw.<init>(command--1:45)
                at $linefd8c230552274eceb0e9701075fd142425.$read$$iw$$iw$$iw.<init>(command--1:47)
                at $linefd8c230552274eceb0e9701075fd142425.$read$$iw$$iw.<init>(command--1:49)
                at $linefd8c230552274eceb0e9701075fd142425.$read$$iw.<init>(command--1:51)
                at $linefd8c230552274eceb0e9701075fd142425.$read.<init>(command--1:53)
                at $linefd8c230552274eceb0e9701075fd142425.$read$.<init>(command--1:57)
                at $linefd8c230552274eceb0e9701075fd142425.$read$.<clinit>(command--1)
                at $linefd8c230552274eceb0e9701075fd142425.$eval$.$print$lzycompute(<notebook>:7)
                at $linefd8c230552274eceb0e9701075fd142425.$eval$.$print(<notebook>:6)
                at $linefd8c230552274eceb0e9701075fd142425.$eval.$print(<notebook>)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
                at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
                at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
                at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
                at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
                at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
                at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
                at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
                at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
                at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:219)
                at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:235)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:887)
                at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:840)
                at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:235)
                at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$11(DriverLocal.scala:526)
                at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:266)
                at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
                at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:261)
                at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:258)
                at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:50)
                at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:305)
                at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:297)
                at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:50)
                at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:503)
                at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:689)
                at scala.util.Try$.apply(Try.scala:213)
                at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:681)
                at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:522)
                at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:634)
                at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:427)
                at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:370)
                at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:221)
                at java.lang.Thread.run(Thread.java:748)

pom.xml

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.6.3</version>
        <relativePath/>
    </parent>

    <dependencies>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.compat.version}</artifactId>
            <version>${spark.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.codehaus.janino</groupId>
                    <artifactId>janino</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.codehaus.janino</groupId>
                    <artifactId>commons-compiler</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.compat.version}</artifactId>
            <version>${spark.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.codehaus.janino</groupId>
                    <artifactId>janino</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.codehaus.janino</groupId>
                    <artifactId>commons-compiler</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.codehaus.janino</groupId>
            <artifactId>janino</artifactId>
            <version>3.0.8</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.3.1</version>
        </dependency>

        <dependency>
            <groupId>com.agent</groupId>
            <artifactId>spring-core</artifactId>
            <version>1.0</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/src/main/resources/spring-core-5.3.15.jar</systemPath>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <dependencies>
                    <dependency>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <version>2.6.3</version>
                    </dependency>
                </dependencies>
                <configuration>
                    <keepDependenciesWithProvidedScope>false</keepDependenciesWithProvidedScope>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <transformers>
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                            <resource>META-INF/spring.handlers</resource>
                        </transformer>
                        <transformer
                                implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                            <resource>META-INF/spring.factories</resource>
                        </transformer>
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                            <resource>META-INF/spring.schemas</resource>
                        </transformer>
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                        <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>com.ApplicationAgent</mainClass>
                        </transformer>
                    </transformers>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>


</project>

Answer 1

Such errors usually occur because Databricks Runtimes (DBR) were including older version of Spring libraries (4.1.4). But starting with DBR 10.4, they are removed, so such errors shouldn't occur. Here is an example of the Spring Boot project for Databricks , that works fine on DBR 10.4.

If you need to use earlier version of DBR, you may consider adding the cluster init script that will remove spring libraries before Spark starts.

Answer 2

yes, Databricks had older version of Spring in their older versions. They fixed it in 10.4. It worked in 10.4.