我无法在 Hadoop 集群中执行 mapreduce 作业

Question

$ hadoop jar /usr/lib/hadoop/hadoop-streaming-2.6.0-cdh5.13.0.jar -file mapper.py -mapper mapper.py -file reducer.py -reducer reducer.py -input /user/cloudera/test.txt -output /user/cloudera/result

I am using this command to execute MapReduce program using Mapper as mapper.py and Reducer as reducer.py

It throws an an error Not a valid JAR:/usr/lib/hadoop/hadoop-streaming-2.6.0-cdh5.13.0.jar

I am using MobaXterm and VMBox & My home directory is /user/cloudera , mapper program location /user/cloudera/mapper.py reducer location /user/cloudera/reducer.py

Answer 1

If you are using cloudera distribution for practice, the JAR will not be available in the mentioned location /usr/lib/hadoop . The hadoop-streaming JAR will be present in /usr/lib/hadoop-mapreduce/

Run with the updated location of the JAR , it should work fine.