Google cloud vertex AI workbench notebook stuck on "starting"

Question

I'm running a Jupyter Notebook on the Vertex AI workbench and getting the error below. It loads for an hour or two, then reverts back to a stopped state. It's a managed notebook, so I can't ssh to the notebook to recover my code, which is all I want to do since there's a lot of work there I haven't backed up (I didn't expect a big Cloud service to randomly fail like this...) Any ideas on how to get this back up and running? I am happy to delete this and start a new instance onc I recover the code on this.

Answer 1

Managed to get it running myself again in meantime.. Make sure your idle timeout is not set too high, as it appears the same timeout value is valid for scenarios like these, which gives you the opportunity to start the notebook again as soon it "gives up" after being idle (in "Starting" state) for too long.

Answer 2

You can always use gcloud if the console fails. Use stop command for managed-notebooks as below:

gcloud notebooks runtimes stop <NOTEBOOK_NAME> --location=<LOCATION>