无法摆脱java.lang.ClassNotFoundException：类org.apache.hadoop.hive.ql.io.RCFileInputFormat

Question

Trying to run a simple mapreduce code that reads data from a RCFile.

I'm running the code using hadoop command :

hadoop jar MRJobRCFile.jar MRJobRCFile <inputRCfile> <outputfile>  

Inspite of adding hive-exec jar to hadoop classpath, getting this error.

export HADOOP_CLASSPATH=/opt/cmr/hadoopinstall/hive-0.10.0-cdh4.4.0/lib/hive-exec-0.10.0-cdh4.4.0.jar

How else can I add the jar?

Tried checking the jars loaded in jvm using verbose:class :

[Loaded org.apache.hadoop.hive.ql.io.RCFileInputFormat from file:/opt/cmr/hadoopinstall/hive-0.10.0-cdh4.4.0/lib/hive-exec-0.10.0-cdh4.4.0.jar]

RCFileInputFormat is being loaded by JVM.

Any idea how to proceed on this issue ? error:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.RCFileInputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1649)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:620)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.ql.io.RCFileInputFormat not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1617)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java

Answer 1

Resolved by adding the hive-exec jar to HADOOP_CLASSPATH and to the distributed cache.

Adding the jar to distributed cache is to make them available to the remote map and reduce task JVM's. And adding in HADOOP_CLASSPATH is for the client JVM (created by hadoop jar command).