Docker 容器上帶有 GPU 的 Pytorch 錯誤 - 未檢測到支持 CUDA 的設備

Question

我正在嘗試在我的 Docker 容器上使用帶有 GPU 的 Pytorch。

1. 在主機上 -我安裝了 nvidia-docker、CUDA 驅動程序等

這是主機的 nvidia-smi 輸出：

    Fri Mar 20 04:29:49 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P8    28W / 149W |     16MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1860      G   /usr/lib/xorg/Xorg                            15MiB |
+-----------------------------------------------------------------------------+

2. 在 Docker 容器上（應用程序的 Dockerfile - 下面的 Docker Compose 文件） -

FROM ubuntu:latest
FROM dsksd/pytorch:0.4
#FROM nvidia/cuda:10.1-base-ubuntu18.04 
#FROM nablascom/cuda-pytorch
#FROM nvidia/cuda:10.0-base

RUN apt-get update -y --fix-missing
RUN apt-get install -y python3-pip python3-dev build-essential
RUN apt-get install -y sudo curl
#RUN sudo apt-get install -y nvidia-container-toolkit
#RUN apt-get install -y curl python3.7 python3-pip python3.7-dev python3.7-distutils build-essential
#RUN apt-get install -y curl
#RUN apt-get install -y sudo
#RUN curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
#RUN sudo dpkg -i cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
#RUN sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
#RUN sudo apt-get install cuda -y
#----------
# Add the package repositories
#RUN distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
#RUN curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
#RUN curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
#RUN sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
#RUN sudo systemctl restart docker
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/usr/local/cuda-10.1/compat/
ENV PYTHONPATH $PATH
#----------
ENV LC_ALL=mylocale.utf8
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT ["python3"]
EXPOSE 5000
CMD ["hook.py"]

當我嘗試在 GPU 上運行我的代碼時，我遇到了：

>>> torch.cuda.current_device()
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=50 error=100 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 386, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 193, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50

我使用以下命令調用容器： docker-compose up --build

這是我的 docker-compose.yaml 文件：

version: '3.6'
services:
  rdb:
    image: mysql:5.7
    #restart: always
    environment:
      MYSQL_DATABASE: 'c_rdb'
      MYSQL_USER: 'user'
      MYSQL_PASSWORD: 'password'
      MYSQL_ROOT_PASSWORD: '123123'
    #ports:
    #  - '3306:3306'
    #expose:
    #  - '3306'
    volumes:
      - rdb-data:/var/lib/mysql
      - ./init-db/init.sql:/docker-entrypoint-initdb.d/init.sql
  mongo:
    image: mongo
    #restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: 12312323
      MONGO_INITDB_DATABASE: chronicler_ndb
    volumes:
      - ndb-data:/data/db
      - ./init-db/init.js:/docker-entrypoint-initdb.d/init.js
    ports:
      - '27017-27019:27017-27019'
  mongo-express:
    image: mongo-express
    #restart: always
    depends_on:
        - mongo
        - backend
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: rooer
      ME_CONFIG_MONGODB_ADMINPASSWORD: 123123
  redis:
    image: redis:latest
    command: ["redis-server", "--appendonly", "yes"]
    hostname: redis
    #ports:
    #  - "6379:6379"
    volumes:
      - cache-data:/data
  backend:
    build: ./app
    ports:
     - "5000:5000"
    volumes:
     - backend-data:/code
    links: 
     - rdb
     - redis

volumes:
  rdb-data:
    name: c-relational-data
  ndb-data:
    name: c-nosql-data
  cache-data:
    name: redis-data
  backend-data:
    name: backend-engine

Answer 1

它需要runtime選項，但是，運行時選項在撰寫文件格式 3 中不可用。所以有一些選項

1. 將您的撰寫文件版本降級到 2，如下所示：

version: 2
  backend:
    build: ./app
    ports:
     - "5000:5000"
    volumes:
     - backend-data:/code
    links: 
     - rdb
     - redis
    runtime: nvidia

2. 或者，使用帶有--runtime=nvidia參數的--runtime=nvidia docker run手動運行容器

另外我建議使用由 nvidia 構建的圖像而不是ubuntu:latest

---

有關更多信息，您可以在此處閱讀問題