How to reset GPU on keras / tensorflow hang?

Question

Sometimes I have to kill my python application which use GPU with Keras or Tensorflow and after that I can't run them anymore. This is probably because GPU is still used by something.

How to free GPU by force without machine reboot?

---

I tried the following shell script

$ cat ~/bin/nvidia-reset
#!/bin/sh

sudo rmmod nvidia_uvm

sudo rmmod nvidia_drm

sudo rmmod nvidia_modeset

sudo rmmod nvidia

sudo nvidia-smi

But often it is unable to do the job saying nvidia_uvm is busy.

Answer 1

尝试这个：

keras.backend.clear_session()