在Hortonworks Hadoop（AWS EC2）上访问WebHDFS

Question

I'm facing an issue with the WebHDFS access on my Amazon EC2 machine. I have installed Hortonworks HDP 2.3 btw.

I can retrieve the file status from my local machine in the browser (chrome) with following http request:

http://<serverip>:50070/webhdfs/v1/user/admin/file.csv?op=GETFILESTATUS

This works fine but if I try to open the file with ?op=OPEN , then it redirects me to the private DNS of the machine, which I cannot access:

http://<privatedns>:50075/webhdfs/v1/user/admin/file.csv?op=OPEN&namenoderpcaddress=<privatedns>:8020&offset=0

I also tried to get access to WebHDFS from the AWS machine itself with this command:

[ec2-user@<ip> conf]$ curl -i http://localhost:50070/webhdfs/v1/user/admin/file.csv?op=GETFILESTATUS
curl: (7) couldn't connect to host

Does anyone know why I cannot connect to localhost or why the OPEN on my local machine does not work? Unfortunately I couldn't find any tutorial to configure the WebHDFS for a Amazon machine.

Thanks in Advance

Answer 1

What happens is that the namenode redirects you to the datanode. Seems like you installed a single-node cluster, but conceptually the namenode and datanode(s) are distinct, and in your configuration the datanode(s) live/listen on the private side of your EC2 VPC.

You could reconfigure your cluster to host the datanodes on the public IP/DNS (see HDFS Support for Multihomed Networks ), but I would not go that way. I think the proper solution is to add a Know gateway , which is a specialized component for accessing a private cluster from a public API. Specifically, you will have to configure the datanode URLs, see Chapter 5. Mapping the Internal Nodes to External URLs . The example there seems spot on for your case:

For example, when uploading a file with WebHDFS service:

• The external client sends a request to the gateway WebHDFS service.

• The gateway proxies the request to WebHDFS using the service URL.

• WebHDFS determines which DataNodes to create the file on and returns the path for the upload as a Location header in a HTTP redirect, which contains the datanode host information.

• The gateway augments the routing policy based on the datanode hostname in the redirect by mapping it to the externally resolvable hostname.

• The external client continues to upload the file through the gateway.

• The gateway proxies the request to the datanode by using the augmented routing policy.

• The datanode returns the status of the upload and the gateway again translates the information without exposing any internal cluster details.