在Hive驱动程序连接线上的Hadoop中由ClassNotFoundException引起的NoClassDefFoundError？

Question

Typically I start by Googling for a solution, but this error does not seem to have occurred before.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/Shell
    at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:906)
    at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:237)
    at org.apache.hive.jdbc.HiveConnection.isHttpTransportMode(HiveConnection.java:221)
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:138)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:123)
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
    at java.sql.DriverManager.getConnection(DriverManager.java:571)
    at java.sql.DriverManager.getConnection(DriverManager.java:215)
    at com.merck.ghh.ingestion.HiveTableSetup.tableSetup(HiveTableSetup.java:31)
    at com.merck.ghh.ingestion.HiveTableSetup.main(HiveTableSetup.java:546)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.Shell
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 10 more

The following code causes this error, specifically on the Connection declaration line:

try {     
Class.forName(driverName);
     Connection connection = DriverManager.getConnection("jdbc:hive2://localhost:10000/default", "hive", "");
     Statement statement = connection.createStatement();
} catch (ClassNotFoundException exception) {
      exception.printStackTrace();
}

Other questions seem to point to this being related to not including hadoop-core in the dependencies, but hadoop-core does not seem to exist in the Hadoop 2.X. I am specifically working in Hadoop 2.1.0.2.0.5.0-67 with Hive 0.12.0. Before this error I was having issues bc my Hive dependencies had gotten destroyed at some point, but I put those back in and this error happened next. I'm wondering if it's not something as simple as forgetting a dependency, but I'm not finding which dependency I might be missing.

Any help with this is greatly appreciated.

Answer 1

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.Shell  

As you are using Hadoop 2.X. you need to download hadoop-common-2.1.0-beta.jar OR
hadoop-common-2.0.0-cdh4.4.0.jar and to class path to avoid ClassNotFoundException .

You are right earlier hadoop-core-0.20.2-737.jar was used.

For more visit http://grepcode.com/

Answer 2

Try copying the JDBC connector jar into the $HIVE_HOME/lib/ folder. I faced a similar issue and copying the jar file worked for me.