EC2上的Hadoop配置：为什么公共DNS有效但公共IP不起作用？

Question

I am setting up Hadoop on EC2. For the fs.default.name , fs.defaultFS , if I use the DNS for the name node, everything works. But if I use the public IP, name node cannot start with the except Problem binding to [54.210.86.207:9000] java.net.BindException: Cannot assign requested address; . Any explanation why the public DNS works but not the public IP?

  <property>
    <name>fs.default.name</name>
    <value>hdfs://54.210.86.207:9000</value>
    <!-- <value>hdfs://ec2-54-210-86-207.compute-1.amazonaws.com:9000</value> -->
  </property>

  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://54.210.86.207:9000</value>
    <!-- <value>hdfs://ec2-54-210-86-207.compute1.amazonaws.com:9000</value> -->
 </property>  

Answer 1

Inside VPC, a DNS query of your instance's public IP address's public hostname does something useful -- and documented -- but otherwise potentially unexpected: it resolves to the machine's private IP address. Queries from outside, of course, resolve to the public address.

We resolve a public DNS hostname to the public IPv4 address of the instance outside the network of the instance, and to the private IPv4 address of the instance from within the network of the instance.

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html

That's exactly why this works.

The public IP address of a machine inside EC2 is not actually bound to the IP stack of the instance's OS -- only the private IP address is. The Internet Gateway handles the 1:1 translation between public and private addresses as traffic traverses it in the way to and from the instance.

And, this is desirable behavior, because you don't want instances talking to other instances (or to themselves!) using their public addresses, since you pay a transport charge for data that hairpins out and back on through the Internet Gateway. Within a single availability zone, you don't pay for traffic between instances when the private IP address is used. Additionally, when two instances communicate with each other via public IP, the identity of the source security group is necessarily lost, which means you can't use the source security-group-id for ingress control.

Answer 2

I haven't worked for hadoop for a while, but when I was working with it, there was no way to reference the namenode or datanodes with only an IP (I think it had something to do with discouraging users to hard code IP addresses in case of failures, but I could be wrong). It's a bad practice to use raw IP addresses, but having this limitation does make messing around with hadoop harder. This is the closest thing I can find to a real source: https://issues.apache.org/jira/browse/HADOOP-685

To get around it, you could either host your own dns server (dnsmasq) and us that or add entries to /etc/hosts of all the machines to make 'fake' names, and use those in the configuration instead. Sorry I don't have a better solution for you...