使用hadoop流命令運行時map和reduce失敗

Question

當我在沒有hadoop流命令的情況下運行時，我的python映射器和reducer代碼運行正常

hadoop fs -cat /user/root/myinput/testfile3_node.csv | ./mapper_1.py | sort | ./reducer_1.py

當我使用hadoop流命令運行代碼時，它失敗了

hadoop jar /usr/iop/current/hadoop-mapreduce-client/hadoop-streaming.jar -mapper ./mapper_1.py -reducer ./reducer_1.py -file ./mapper_1.py -file ./reducer_1.py -input /user/root/myinput/testfile3.csv -output /user/root/myoutput/indexing_output1

輸出：

簡單的command_running屏幕截圖。 Hadoop Streaming jar命令的屏幕截圖。

Answer 1

嘗試在-mapper和-reducer參數上不使用./（確保您在正確的目錄中），並且也不需要-files：

hadoop  jar /usr/iop/current/hadoop-mapreduce-client/hadoop-streaming.jar \
    -mapper mapper_1.py \
    -reducer reducer_1.py \
-input /user/root/myinput/testfile3.csv -output /user/root/myoutput/indexing_output1   

這是Apache Hadoop文檔：

https://hadoop.apache.org/docs/r1.2.1/streaming.html