SSH error when installing Google Dataproc with Jupyter

Question

I have installed a Google DataProc Cluster with Jupyter ( https://cloud.google.com/dataproc/tutorials/jupyter-notebook ). By using the following command:

gcloud dataproc clusters create reco-test --zone=europe-west1-d  --master-machine-type n1-standard-2 --master-boot-disk-size 100 --num-workers 2 --worker-machine-type n1-standard-2 --worker-boot-disk-size 50 --project *project* --bucket *bucket* --initialization-actions  gs://dataproc-initialization-actions/jupyter/jupyter.sh`

Witch worked as a charm. Afterwards I am trying to do the SSH forwarding, but errors appear:

gcloud compute ssh --zone=europe-west1-d --ssh-flag="-D 10000" --ssh-flag="-N" --ssh-flag="-n" "reco-test-m" &

The error that was given can be find in the picture below as Error 1.

If I remove the "-D"

gcloud compute ssh --zone=europe-west1-d --ssh-flag="10000" --ssh-flag="-N" --ssh-flag="-n" "reco-test-m" &

The error that was given can be find in the picture below as Error 2.

If I delete "--ssh-flag= "-n""

gcloud compute ssh --zone=europe-west1-d --ssh-flag="10000" --ssh-flag="-N" --ssh-flag="-n" "reco-test-m" &`

The error that was given can be find in the picture below as Error 3.

Is there somebody who can help me resolving this problem?

enter image description here

Answer 1

Looks like you're hitting this known issue about the way the arguments are parsed in windows: https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/issues/22

Try splitting the -D 10000 into two separate flags:

--ssh-flag="-D" --ssh-flag="10000"

Additionally, the -n flag is only necessary for running it in the background, and doesn't appear supported in Windows. In general it may be more convenient to keep it running in the foreground anyways, so you can easily ctrl+C the the process to kill the tunnel when you're done using it without dealing with background processes (though running it in the background will still terminate automatically when the cluster is deleted).

Update: Since splitting the flag also works in posix systems, I've updated the documentation to now use the separate --ssh-flag arguments and also better explain leaving off the -n flag when running in the foreground.