Error while running maven jar file

Question

I created maven jar file from eclipse. Getting following error while running jar.

Error:

 A JNI error has occurred, please check your installation and try again

    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataOutputStream
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Unknown Source) 
    at java.lang.Class.privateGetMethodRecursive(Unknown Source)

Please guide whether any dependencies need to be added in pom.xml

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>utd.bigdata</groupId>
  <artifactId>hadoop1</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>hw1</name>
  <dependencies>
  <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.4.1</version>
  </dependency>
   <dependency>
     <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.3</version>
        <scope>compile</scope>
       </dependency>
  <!--  <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-auth</artifactId>
    <version>2.4.1</version>
</dependency>-->
  </dependencies>
  <build>
    <plugins>
        <plugin>
            <!-- Build an executable JAR -->
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.4</version>
            <configuration>
                <archive>
                    <manifest>
                        <mainClass>bigdata.UploadHadoop</mainClass>
                        <classpathPrefix>classes/lib</classpathPrefix>
                    </manifest>
                </archive>
            </configuration>
        </plugin>
    </plugins>
</build>
</project>

Also is there any error in pom.xml?Please suggest any code changes

Answer 1

A regular .jar archive does not contain its dependencies at runtime, for this you can use the maven-assembly-plugin to create a fat executable Jar:

https://maven.apache.org/plugins/maven-assembly-plugin/usage.html

<!-- Create a fat Jar -->
<plugin>
  <artifactId>maven-assembly-plugin</artifactId>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>single</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <archive>
      <manifest>
        <mainClass>bigdata.UploadHadoop</mainClass>
      </manifest>
    </archive>
    <descriptorRefs>
      <descriptorRef>jar-with-dependencies</descriptorRef>
    </descriptorRefs>
  </configuration>
</plugin>

Answer 2

As an alternative to a "fat jar", the hadoop jar runner also supports the "jar of jars" format. It's worth noting that it's not standard for a jar, but since it's supported in hadoop, here it is: How do I put all required JAR files in a library folder inside the final JAR file with Maven?

This format is a normal jar, but with a lib/ folder containing all dependency jars.

Answer 3

If you use maven-jar-plugin to build an executable jar, please read the document. http://maven.apache.org/shared/maven-archiver/examples/classpath.html#Make

First, add maven-jar-plugin in pom.xml

<build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        ...
        <configuration>
          <archive>
            <manifest>
              <addClasspath>true</addClasspath>
              <mainClass>bigdata.UploadHadoop</mainClass>
              <classpathPrefix>classes/lib</classpathPrefix>
            </manifest>
          </archive>
        </configuration>
        ...
      </plugin>
    </plugins>
  </build>

Second, It doesn't attach the dependency jars to the project jar. So you must be sure the dependency jars(hadoop-client, hadoop-common) are in classes/lib by yourself.

At last, I suggest that you use maven-shade-plugin to build an executable jar(fat jar) http://maven.apache.org/plugins/maven-shade-plugin/examples/executable-jar.html It will attach the dependency jars to the result shade jar by itself. You also use maven-assembly-plugin to build an executable jar, but not recommended.

<build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>bigdata.UploadHadoop</mainClass>
                </transformer>
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>