How to set HADOOP_CLASSPATH for using the local filesystem with a local job runner?

Question

How to set the HADOOP_CLASSPATH for using the local filesystem with a local job runner? How to set the input and output path from local directories?

ClassNotFoundException arises for mapper and reducer classes when I try to run with the following command.

hadoop WordCount input/sample.txt output

current value is:

: hadoop classpath

/usr/local/hadoop/hadoop-3.2.1/etc/hadoop:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/common/lib/*:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/common/*:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/hdfs:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/hdfs/lib/*:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/hdfs/*:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/mapreduce/*:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/yarn:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/yarn/lib/*:/usr/local/hadoop/hadoop-3.2.1/share/hadoop/yarn/*:/usr/local/hadoop/hadoop-3.2.1/etc/hadoop/usr/local/hadoop/hadoop-3.2.1/share/hadoop/common/*.jar

versions:
Apache hadoop-3.2.1 ,
openjdk 11.0.5.

please help. It is useful for debugging. Thanks in advance.

Answer 1

I would believe if you had FileNotFound Exception, but your classpath seems fine, so I have a hard time seeing you would get ClassNotFound Exception

Although, this path seems wrong /usr/local/hadoop/hadoop-3.2.1/etc/hadoop/usr/local/hadoop/hadoop-3.2.1/share/hadoop/common/*.jar .

I would suggest moving all files under hadoop-3.2.1 up into /usr/local/hadoop , or at the very least, rename hadoop-3.2.1 directory to just /usr/local/hadoop/3.2.1/

By default, Hadoop jobs use file:// paths as your fs.defaultFS (defined in core-site.xml)

Otherwise, if you have changed that to use hdfs:// , then you can still use local files like so

hadoop fs -ls file://

To run jobs, I would suggest using yarn jar , not hadoop <name> . And you need to shade your Java application into an uber-jar, or use the existing hadoop-examples JAR to run WordCount