java.lang.NoSuchMethodError in Flink

Question

I trying to read the a file using :

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");

I get following error:

java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found

I'm using flink version 1.3.2, java version "1.8.0_91"

Answer 1

There is a conflict with dependencies. Apache Flink loads many classes by default into its classpath.

Please read this article https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html the last section

Resolving Dependency Conflicts with Flink using the maven-shade-plugin

Apache Flink loads many classes by default into its classpath. If a user uses a different version of a library that Flink is using, often

IllegalAccessExceptions

or

NoSuchMethodError

are the result.

So, I suggest to play with your pom.xml and use maven-shade-plugin and add correct relocation, as we have in example

<relocation>
  <pattern>org.codehaus.plexus.util</pattern>
  <shadedPattern>org.shaded.plexus.util</shadedPattern>
  <excludes>
    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
  </excludes>
</relocation>

Answer 2

Are you getting this error in IntelliJ or Dashboard, if you are getting this error in IntelliJ then make sure you use the same Flink version in your pom.xml and also add dependency shading in the build like this

 <build>

        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>

                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <compilerId>jdt</compilerId>
                </configuration>

                <dependencies>
                    <dependency>
                        <groupId>org.eclipse.tycho</groupId>
                        <artifactId>tycho-compiler-jdt</artifactId>
                        <version>0.21.0</version>
                    </dependency>
                </dependencies>
            </plugin>


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>org.codehaus.plexus.util</pattern>
                                    <shadedPattern>org.shaded.plexus.util</shadedPattern>
                                    <excludes>
                                        <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                                        <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                                    </excludes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>


        </plugins>


</build>>

make sure to run maven clean install in the terminal after you make changes . On the other hand, If you are having this issue only in Dashboard not in intelliJ , then have a look here

Answer 3

I faced the similar issue, for me the problem was flink minor version mismatch. My local flink cluster was running flink-1.8.0 and my code expected the version to be flink-1.8.3. Switching to newer version solved this issue.

Answer 4

您需要检查您的构建路径，确保库在那里并正确导入它们

Answer 5

One possible cause for error "java.lang.NoSuchMethodError" is when we use different version of flink then what we have installed on our system. For me, I have Flink 1.3.2 and the version I was using was 1.1.1 . So I updated my pom file to have same version.

Answer 6

For the ones who're stuggling with the above issue, while using Flink 1.3.2 , here's the entire pom which I was able to successfully build:

<build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.4.1</version>
            <configuration>
               <!-- get all project dependencies -->
               <descriptorRefs>
                  <descriptorRef>jar-with-dependencies</descriptorRef>
               </descriptorRefs>
               <!-- MainClass in mainfest make a executable jar -->
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
            <executions>
               <execution>
                  <id>make-assembly</id>
                  <!-- bind to the packaging phase -->
                  <phase>package</phase>
                  <goals>
                     <goal>single</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
               <archive>
                  <manifest>
                     <mainClass>FlinktoLambda</mainClass>
                  </manifest>
               </archive>
            </configuration>
         </plugin>
         <!--added newly-->
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
               <source>1.8</source>
               <target>1.8</target>
               <compilerId>jdt</compilerId>
            </configuration>
            <dependencies>
               <dependency>
                  <groupId>org.eclipse.tycho</groupId>
                  <artifactId>tycho-compiler-jdt</artifactId>
                  <version>0.21.0</version>
               </dependency>
            </dependencies>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.0</version>
            <executions>
               <execution>
                  <phase>package</phase>
                  <goals>
                     <goal>shade</goal>
                  </goals>
                  <configuration>
                     <transformers>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                           <mainClass>FlinktoLambda</mainClass>
                        </transformer>
                        <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                           <resource>reference.conf</resource>
                        </transformer>
                     </transformers>
                     <relocations>
                        <relocation>
                           <pattern>org.codehaus.plexus.util</pattern>
                           <shadedPattern>org.shaded.plexus.util</shadedPattern>
                           <excludes>
                              <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                              <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                           </excludes>
                        </relocation>
                     </relocations>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
   <name>my-app</name>
   <url>http://maven.apache.org</url>
   <dependencyManagement>
      <dependencies>
         <dependency>
            <groupId>FlinktoLambda</groupId>
            <artifactId>my-app</artifactId>
            <exclusions>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-remote_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-actor_2.10</artifactId>
               </exclusion>
               <exclusion>
                  <groupId>com.typesafe.akka</groupId>
                  <artifactId>akka-slf4j_2.10</artifactId>
               </exclusion>
            </exclusions>
         </dependency>
      </dependencies>
   </dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-table_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-java</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-java_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-clients_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
      <dependency>
         <groupId>org.apache.flink</groupId>
         <artifactId>flink-streaming-scala_2.10</artifactId>
         <version>1.3.2</version>
      </dependency>
   </dependencies>

Please change your main class accordingly in the shade plugin.

Answer 7

Another source of this error could be a mismatch of scala version between Flink and your application code (or your application dependencies that uses scala).

For instance, in my case, I was using Flink 1.7.1 and I had to update my scala dependencies from 2.11 to 2.12; I updated the artifcatId of the concerned dependencies as follows: from flink-scala_2.11 to flink-scala_2.12 , flink-table_2.11 to flink-table_2.12 , etc.

See here for more info.