[英]python streaming mapreduce job on hadoop failed - missing log4j?
我嘗試在Ubuntu 15.10上安裝的hadoop 2.7.1上運行python wordcount,但出現錯誤:
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
另外,我在終端中收到RunTimeException錯誤,並且信息流傳輸失敗並且沒有輸出文件。
我發現一些線程說log4j.properties和log4j.xml可能丟失了,還列舉了log4j.properties應該包含的內容,我嘗試了一個例子,但沒有成功。 我在哪里可以找到Hadoop目錄中的文件(如果可以找到它們),或者如何使用正確的配置來創建它們?
從這里獲取用於mapper和reducer for wordcount的代碼,並且在
input.txt|./mapper.py|sort|./reducer.py
但是,我嘗試過幾次在hadoop上運行它,但失敗了。 當將python文件復制到hdfs以及將它們放在本地文件系統上時,我使用了不同的命令進行嘗試:此命令不起作用:
hadoop hadoop-streaming-2.7.1.jar -mapper /user/mapper.py -reducer /user/reducer.py -input/input_file.txt -output /user/output
也不是這個:
hadoop hadoop-streaming-2.7.1.jar -mapper "python /user/mapper.py" -reducer "python /user/reducer.py" -input/input_file.txt -output /user/output
這確實起作用了(本地文件系統中的python文件):
hadoop hadoop-streaming-2.7.1.jar -mapper "python /home/user_name/Documents/mapper.py" -reducer "python /home/user_name/Documents/reducer.py -input /user/input_file.txt -output /user/output
所有文件都具有正確的權限。
在標准開始之后的輸出如下:
16/02/15 09:47:48 INFO mapreduce.Job: map 0% reduce 0%
16/02/15 09:48:05 INFO mapreduce.Job: Task Id : attempt_1455529218252_0001_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: configuration exception
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)
... 22 more
Caused by: java.io.IOException: Cannot run program "/user/mr/mapper.py": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
... 23 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 24 more
還有很多,但是最終輸出是關於流作業失敗的:
16/02/15 09:49:07 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=7
Killed map tasks=1
Launched map tasks=8
Other local map tasks=6
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=135543
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=135543
Total vcore-seconds taken by all map tasks=135543
Total megabyte-seconds taken by all map tasks=138796032
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
16/02/15 09:49:07 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!
從hdfs調用時python代碼不起作用的原因可能是什么?
您應該只提供本地 python文件的名稱作為-mapper
和-reducer
參數。 它們不必位於HDFS上,也不必在命令行中提供字符串來執行腳本。
您還需要為每個腳本提供一個-file
參數。 嘗試使用
hadoop hadoop-streaming-2.7.1.jar -file /home/user_name/Documents/mapper.py -file /home/user_name/Documents/reducer.py -mapper /home/user_name/Documents/mapper.py -reducer /home/user_name/Documents/reducer.py -input /input_file.txt -output /user/output
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.