[英]Hadoop getting Connection Refused exception while starting job from Spring batch admin
我試圖從spring Admin觸發hadoop mapreduce作業,但出現錯誤。 Spring Admin在wasce上運行。
作業配置:
<hdp:configuration>
fs.defaultFS=hdfs://localhost:8020
mapred.job.tracker=localhost:8021
</hdp:configuration>
<hdp:job id="mr-my-job"
input-path="/data/input/"
output-path="/data/output/"
jar-by-class="org.test.Main"
mapper="org.test.Test1$Map"
combiner="org.test.Test1$Combiner"
reducer="org.test.Test1$ReduceFromCombiner" />
請指教。 當我通過打包一個jar並使用hadoop命令運行此作業時,它工作正常,但在通過Admin運行時卻出現錯誤。 sudo -u hdfs hadoop jar test.jar org.Main
Caused by: java.net.ConnectException: Call From <server>/<server_ip> to localhost:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1413)
at org.apache.hadoop.ipc.Client.call(Client.java:1362)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:231)
at org.apache.hadoop.mapred.$Proxy203.getStagingAreaDir(Unknown Source)
at org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1340)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:954)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:948)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:948)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:582)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:612)
at org.springframework.data.hadoop.mapreduce.JobExecutor$2.run(JobExecutor.java:199)
... 24 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1461)
at org.apache.hadoop.ipc.Client.call(Client.java:1380)
我們在hadoop上有一個與此相關的特殊頁面。 請遵循您看到的堆棧跟蹤中的鏈接: http : //wiki.apache.org/hadoop/ConnectionRefused
我可以說, 我已經花了很多精力來嘗試改善連接診斷,包括目標主機和端口的報告,帶有調試建議的URL列表以及這些頁面的編寫和維護。 當人們不遵循鏈接時,這總是讓我感到非常難過,因為我覺得我的精力已經浪費了。 請在堆棧跟蹤中看到這樣的URL時-至少要粗略地看一下所引用的URL。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.