Cannot SSH Dataproc Master in Cluster

Question

I've been trying to create a cluster in Dataproc using as initialization script the jupyter repository .

But when I try to ssh to the master so to be able to access the Jupyter interface running this command:

gcloud compute ssh --zone=zone_name \
                   --ssh-flag="-D 10000" \
                   --ssh-flag="-N" \
                   --ssh-flag="-n" "cluster1-m" &

I get the error:

Permission denied (publickey). ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

I could confirm that all ssh keys are created normally. I tried this other option then:

gcloud compute ssh --zone=zone_name \
                   --ssh-flag="-D 10000" \
                   --ssh-flag="-N" \
                   --ssh-flag="-n" "will@cluster1-m" &

Which seems to work as I can ssh into the instance but now I get the error:

bind: Cannot assign requested address channel_setup_fwd_listener_tcpip: cannot listen to port: 10000 Could not request local forwarding.

For creating the cluster I used:

gcloud dataproc clusters create $CLUSTER_NAME \
--metadata "JUPYTER_PORT=8124,JUPYTER_CONDA_PACKAGES=numpy:pandas:scikit-learn:jinja2:mock:pytest:pytest-cov" \
--initialization-actions \
    gs://dataproc-initialization-actions/jupyter/jupyter.sh \
--bucket $BUCKET_NAME

and I'm running this in a docker image Debian 8.9 (jessie).

If you need any extra information please let me know.

Answer 1

If you verified you can do a normal SSH into the cluster, then if you're just getting the bind: Cannot assign requested address error it probably means you have another SSH session with local port forwarding already on your current machine using port 10000 already. If you see the bind error, you should always first try a different local port, like -D 12345 . You can use top or your task manager to check if you have a hanging ssh -D command somewhere still running and occupying port 10000.