Tensorflow Colab：运行时已断开连接到运行时已超时

Question

How come after 2 hours of running a model, I get a popup window saying:

    Runtime disconnected

    The connection to the runtime has timed out.


                     CLOSE             RECONNECT

I had restarted my runtime and thought I have 12 hours to train a model. Any Idea how to avoid this? My other question: Is it possible to find out the time left for runtime to get disconnected using a TF or Python API?

Answer 1

Runtime gets disconnected when the notebook goes to "idle" mode for a time greater than 90 minutes. This is an unofficial number, as google colab has no official release about this. This is how google colab gets away with it by answering cheekily:

An extract from the Official Colab FAQ

Where is my code executed? What happens to my execution state if I close the browser window?

Code is executed in a virtual machine dedicated to your account. Virtual machines are recycled when idle for a while, and have a maximum lifetime enforced by the system.

So to avoid this, keep your browser open and don't let your system sleep for a time greater than 90 minutes.

This also means if you happen to close your browser within the 90 minutes, then if you reopen the notebook within 90 minutes you will still have all your running processes and session variables intact!

Also, note that currently you can run a notebook for a maximum of 12 hours. (in the "non-idle" state of course).

To answer your second question, this "idle state" stuff is a colab thing. So I don't think TF or Python will have anything to do with it.

So it is good practise to have your models saved into a folder periodically. This way, in the unfortunate event of your runtime getting disconnected, your work will not be lost. And you can simply restart your training from the latest saved model!

PS: I got the number 90 minutes from an experiment done by a fellow user