Spark-线程中的异常 java.lang.NoSuchMethodError

Question

I am trying to score a model from pmml file using pmml4s library. Every time I submit the job in Spark I get the following error:

20/05/13 23:30:10 ERROR SparkSubmit: org.apache.spark.sql.types.StructType.names(). 
[Ljava/lang/String;
   java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.names(). 
    [Ljava/lang/String;
    at org.pmml4s.spark.ScoreModel.transform(ScoreModel.scala:56)
    at com.aexp.JavaPMML.main(JavaPMML.java:24)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)

Following is my code sample:

ScoreModel model = ScoreModel.fromFile(args[0]);
SparkConf conf = new SparkConf();
SparkSession spark = SparkSession.builder().config(conf).getOrCreate();
Dataset<?> df = spark.read().format("csv")
                     .option("header", "true")
                     .option("inferSchema", "true")
                     .load(args[1]);

Dataset<?> scoreDf = model.transform(df);

Following is the pom file that I am using:

<dependencies>
    <dependency>
        <groupId>org.pmml4s</groupId>
        <artifactId>pmml4s-spark_2.11</artifactId>
        <version>0.9.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
          <artifactId>spark-core_2.11</artifactId>
          <version>2.3.2</version>
        </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.3.2</version>
    </dependency>
</dependencies>

I have edited my pom file and made the spark version similar still I face the same issue. When I am using Scala, I am facing the same problem. Is there any dependency that I am missing?

Answer 1

Try to use same version of spark libraries. If spark versions are not matching we will be getting NoSuchMethodError issue in many places as those methods might have modified or removed in latest versions.

Answer 2

The error is caused by the PMML4S-Spark used the method names of StructType, which is introduced since Spark 2.4. Now it has been fixed in the latest PMML4S-Spark 0.9.5 . Please, update your pom file to use the new version:

    <dependency>
        <groupId>org.pmml4s</groupId>
        <artifactId>pmml4s-spark_2.11</artifactId>
        <version>0.9.5</version>
    </dependency>