当运行Keras / tensorflow-gpu时，GPU崩溃，特别是当时钟速度在0 MHz处空闲时

Question

I'm using Jupyter Notebook to run Keras with a Tensorflow GPU backend. I've done some testing with various dummy models while simultaneously monitoring my GPU usage using MSI Afterburner, GPU-Z, nvidia-smi and Task Manager. My GPU is a GeForce GTX 960M, which has no issues running games. The temperatures are also low when running Keras.

What I've noticed is that the Keras runs fine (eg loading or training a model) in the beginning but whenever Keras is not running anything, the GPU naturally wants to idle from 1097 MHz to 0 MHz and as soon as it does that the GPU crashes. I can see that the "GPU is lost" on NVSMI. I have to then disable and re-enable my GPU in the Device Manager to get it to work.

Does anyone have any idea why this might be happening?

Edit: I can temporarily prevent this from happening for very small programs by using the "allow_growth" feature as follows:

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
set_session(sess)

However, this only works if the operation is really small such that it uses only about 0.1 GB of GPU memory such as loading a model or running a really small model. However, if the program is using memory of even 0.3 GB of memory my GPU crashes since the memory does not go to 0 GB before the clock speed drops to 0 MHz (lower power state).

Answer 1

I was finally able to figure out the issue thanks to someone from another forum. It was a driver issue. The latest drivers provided by Nvidia are causing the issue unlike the old drivers provided by my laptop manufacturer.

Since I was not able to run tensorflow with my old drivers and do more troubleshooting, what I did was download eDrawings Viewer and open up some random assembly drawings I found online. First I tried with the latest Nvidia drivers, and I see that when I manipulate the models, my card is at P0 state but if I don't do anything and let the software idle, my card goes to a lower power state and crashes my GPU. But when I did the same exercise with my ASUS manufacturer-certified drivers (since this software was compatible even with the older drivers unlike TF), my GPU did NOT crash.

What I also discovered was that eDrawings Viewer does not crash even with the latest Nvidia drivers if I go into the Nvidia Control Panel and select "Prefer Maximum Performance" under Power Management Mode. The card stays at P0 state whenever I have the software open even after idling for minutes. Unfortunately, since python.exe does not have a graphical interface, this option does not work for my case. As a workaround, I can still run tensorflow without getting it to crash by running eDrawings Viewer in the background (or really any program that uses a graphical interface), which keeps my card at the P0 State.