Google dataproc: Unable to access spark history page

Question

I created a Google dataproc cluster. After logging into master node I started spark-shell then trying to access spark history page using

http://<external_ip_masternode>:4040

It get redirected to

http://<hostname_mastername>:8088/proxy/application_1487485713573_0002/

Browser is rejecting with error "DNS address could not be found." which is understandable.

Following are VM instance setting

Public IP type Ephermal tcp:4040 opened in firewall ip forwarding Off: Unable to edit this configuration

Following troubleshooting done but did not help

Telnet to :4040 -> Working

Access from Ubantu host/ browser Chrome: Getting redirected and name lookup failure

Access from Ubantu host /browser Firefox: Getting redirected and name lookup failure

Access from Mac OSX host /browser Safari : Getting redirected and name lookup failure

Access from Mac OSX host/ browser chrome : Getting redirected and name lookup failure

Answer 1

To view Hadoop web interfaces in Dataproc, it is recommended to follow the instructions for running an SSH-based SOCKS proxy: https://cloud.google.com/dataproc/docs/concepts/cluster-web-interfaces

If you follow the instructions there, it'll also have you run a separate browser session using your SSH tunnel, and sets hostname resolution to occur on the VM side of the tunnel. That way, all the links in the Hadoop pages will automatically work, since they all reference each other using internal hostnames, and intentionally avoid any dependency on "external IP addresses".

Using the SSH tunnel is also much more secure than opening up firewall rules to visit the unencrypted HTTP traffic directly coming from the Hadoop HTTP servers (if you accidentally open up your firewall rules too broadly, then other people on the internet will be able to access your external IP addresses, and even if you don't, attackers could see your unencrypted web traffic served up by the ApplicationMaster, HistoryServer, etc.).