初始化 NVML 失败：几个小时后 Docker 出现未知错误

Question

I am having interesting and weird issue.

When I start docker container with gpu it works fine and I see all the gpus in docker. However, few hours or few days later, I can't use gpus in docker.

When I do nvidia-smi in docker machine. I see this msg

"Failed to initialize NVML: Unknown Error"

However, in the host machine, I see all the gpus with nvidia-smi. Also, when I restart the docker machine. It totally works fine and showing all gpus.

My Inference Docker machine should be turned on all the time and do the inference depends on server requests. Does any one have same issue or the solution for this problem?

Answer 1

I had the same issue, I just ran screen watch -n 1 nvidia-smi in the container and now it works continuously.

Answer 2

I had the same Error. I tried the health check of docker as a temporary solution. When nvidia-smi failed, the container will be marked unhealth, and restart by restart policy.

Docker-compose Version:

healthcheck:
  test: nvidia-smi || exit 1
  start_period: 60s
  interval: 20s
  timeout: 10s
  retries: 2

Dockerfile Version:

HEALTHCHECK \
    --start-period=60s \
    --interval=20s \
    --timeout=10s \  
    --retries=2 \
    CMD nvidia-smi || exit 1