Google DataProc Spark - getting “permission denied (publickey)” error when trying to SSH to a worker node

Question

small cluster. 1 master, 2 workers. I can access all nodes (master+slave) just fine using gcloud SDK. However, once I access the master node and try to ssh to a slave node, I get "permission denied (publickey)" error. Note that I can ping the node successfully, but SSH does not work.

Answer 1

Dataproc does not install SSH keys between the master and worker nodes, so that is working as intended.

You may be able to use SSH agent forwarding. With something like:

# Add Compute Engine private key to SSH agent
ssh-add ~/.ssh/google_compute_engine
# Forward key to SSH agent of master
gcloud compute ssh --ssh-flag="-A" [CLUSTER]-m
# SSH into worker
ssh [CLUSTER]-w-0

You could also configure SSH keys using an initialization action or use gcloud ssh from the master node (if you gave the cluster the compute.rw scope).