Hadoop 2.6 Connecting to ResourceManager at /0.0.0.0:8032

Question

I´m trying to run the following Spark example under Hadoop 2.6, but I get the following error:

INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 and the Client enters in a loop trying to connect. I´m running a cluster of two machines, one master and a slave.

./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn-cluster \
--num-executors 3 \
--driver-memory 2g \
--executor-memory 2g \
--executor-cores 1 \
--queue thequeue \
lib/spark-examples*.jar \
10

This is the error I get:

15/12/06 13:38:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable  
15/12/06 13:38:29 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032  
15/12/06 13:38:30 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)  
15/12/06 13:38:31 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)   
15/12/06 13:38:32 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)   
15/12/06 13:38:33 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)   
15/12/06 13:38:34 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

jps

hduser@master:/usr/local/spark$ jps

4930 ResourceManager 
4781 SecondaryNameNode 
5776 Jps 
4608 DataNode 
5058 NodeManager 
4245 Worker 
4045 Master 

My /etc/host/

/etc/hosts

192.168.0.1 master 
192.168.0.2 slave 

The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback 

fe00::0 ip6-localnet 

ff00::0 ip6-mcastprefix 

ff02::1 ip6-allnodes 

Answer 1

主机名配置不正确时，主要出现此错误...请检查主机名是否配置正确，与您在Resourcemanager中提到的相同...

Answer 2

I had faced the same problem. I solved it.

Do the Following steps.

1. Start Yarn by using command: start-yarn.sh
2. Check Resource Manager by using command: jps
3. Add the following code to the configuration

<property>
   <name>yarn.resourcemanager.address</name>
   <value>127.0.0.1:8032</value>
</property>

Answer 3

I had also encountered the same issue where I was not able to submit the spark job with spark-submit.

The issue was due to the missing HADOOP_CONF_DIR path while launching the Spark job So, whenever you are submitting the job, set HADOOP_CONF_DIR to appropriate HADOOP CONF directory. Like export HADOOP_CONF_DIR=/etc/hadoop/conf

Answer 4

您需要确保yarn-site.xml位于类路径上，并确保相关属性标记为true元素。

Answer 5

当我运行./bin/yarn-session.sh -n 2 -tm 2000时，类似的导出HADOOP_CONF_DIR = / etc / hadoop / conf对于我在flink上的情况是一个好主意。

Answer 6

As you can see here yarn.resourcemanager.address is calculated based on yarn.resourcemanager.hostname which its default value is set to 0.0.0.0 . So you should configure it correctly.
From the base of the Hadoop installation, edit the etc/hadoop/yarn-site.xml file and add this property.

  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
  </property>

Exucuting start-yarn.sh again will put your new settings into effect.

Answer 7

I have got the same problem. My cause is that the times are not the same between machines since my Resource Manager is not on the master machine. Just one second difference can cause yarn connection problem. A few more seconds difference can cause your name node and date node unable to start. Use ntpd to configure time synchronization to make sure the times are exactly same.