如何使用 docker-compose 设置一个容器以允许它的整个卷访问另一个容器

Question

I have 4 services running using the docker-compose

1) python-api 2) python-model 3) python-celery 4) redis-server .

Flow:

1) python-api gets hit via postman with images and some text as parameters on port 8000.

2) python-api passes the image and data to python-model on port 8001 for some ML predictions.

3) The modified image and response data in JSON format is then passed to python-celery for triggering mails.

Error: python-celery is able to grab hold of images and responses that are being sent by python-model in step3. But it's not able to read image currently

Error log:

========================
python-celery_1      | Received task: classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17]
python-celery_1      | == unique_file_index
python-celery_1      | AANJkaNIJSDHURHQEYRQ(*R
python-celery_1      | /python-model/server/classify_crack/inference/images/202003251237371/202003251237371_0.jpg 64
python-celery_1      | Task classify_crack.tasks.queue_task_v3[d71f976f-b2e7-4b29-9147-35996668de17] raised unexpected: AttributeError("'NoneType' object has no attribute 'shape'",)
python-celery_1      | Traceback (most recent call last):
python-celery_1      |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
python-celery_1      |     R = retval = fun(*args, **kwargs)
python-celery_1      |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
python-celery_1      |     return self.run(*args, **kwargs)
python-celery_1      |   File "/python-api/classify_crack/tasks.py", line 690, in queue_task_v3
python-celery_1      |     save_heat_map_v2(predictions, img_path, _dir, unique_file_index, original_image_index, i, grid_size=grid_size, metadata=metadata, StoredFileLinks=StoredFileLinks, row_stride=row_stride, col_stride=col_stride)
python-celery_1      |   File "/python-api/classify_crack/tasks.py", line 128, in save_heat_map_v2
python-celery_1      |     num_row_splits = int(np.ceil(img.shape[0]/row_stride))
python-celery_1      | AttributeError: 'NoneType' object has no attribute 'shape'

Line inside python-celery code where I'm getting an error:

  img = cv2.imread(img_path)
  print(img_path)
  print(img)
  # print(img_path,grid_size)
  splits = int(np.ceil(img.shape[0]/row_stride))

Here, img_path is a valid path inside the container that is being printed. But I'm not able to read the image as img returns None . and the line splits giving me the above error.

Reason why I'm getting an error:

I'm receiving this error because it is trying to access the folder path: /python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg , but python-celery is not able to access that folder with the name 202003251237371 .

Proof:

I tried using the command:

command: >
      sh -c "ls '/python-model/server/classify_crack/inference/images' &&

inside the docker-compose of both python-model and python-celery services, I get the following outcome while I run all the containers again:

python-model_1       | 201801151543500
python-model_1       | 201801151543500.jpg
python-model_1       | IMG_20190307_184100
python-model_1       | IMG_20190307_184100.jpg
python-model_1       | extracted_input_0_0 (15)
python-model_1       | extracted_input_0_0 (15).jpg
python-model_1       | extracted_input_0_0 (16)
python-model_1       | extracted_input_0_0 (16).jpg
python-model_1       | extracted_input_0_0 (18)
python-model_1       | extracted_input_0_0 (18).jpg
python-model_1       | extracted_input_0_0 (19)
python-model_1       | extracted_input_0_0 (19).jpg
python-model_1       | extracted_input_0_0 (9)
python-model_1       | extracted_input_0_0 (9).jpg
python-model_1       | file
python-model_1       | image (2)
python-model_1       | image (2).png
python-model_1       | 202003251237371
python-model_1       | 202003251237371.jpg
python-model_1       | image_X
python-model_1       | image_X.png
python-model_1       | original
python-model_1       | original.jpg
python-model_1       | original_image_0
python-model_1       | original_image_0.jpg



python-celery_1      | 20013V_Y.JPG
python-celery_1      | extracted_input_0_0 (15).jpg
python-celery_1      | extracted_input_0_0 (16).jpg
python-celery_1      | extracted_input_0_0 (18).jpg
python-celery_1      | extracted_input_0_0 (19).jpg
python-celery_1      | extracted_input_0_0 (4).jpg
python-celery_1      | 202003251237371.jpg

Now clearly, python-celery cannot display folder 202003251237371 with the image name 202003251237371.jpg , which I could see in python-model .

How to tackle this scenario and allow python-celery to access such image folders?

docker-compose

version: "3"
networks:
  app-tier:
    driver: bridge

volumes:
  app-volume: {}

services:
  python-api-celery: &python-api-celery
    build:
      context: /Users/AjayB/Desktop/python-api/
    networks:
      - app-tier
    volumes:
      - app-volume:/python-model/server/classify_crack/:rw
    environment:
      - PYTHON_API_ENV=development

    command: >
      sh -c "python manage.py makemigrations &&
             python manage.py migrate"

  python-api: &python-api
    <<: *python-api-celery
    ports:
      - "8000:8000"
    command: >
      sh -c "python manage.py runserver 0.0.0.0:8000"

  python-celery: &python-celery
    <<: *python-api-celery
    depends_on:
      - redis
    links:
      - python-model

    command: >
      sh -c "ls '/python-model/server/classify_crack/inference/images' &&
             celery -A server worker -l info"

  redis:
    image: redis:5.0.8-alpine
    hostname: redis
    networks:
          - app-tier
    expose:
      - "6379"
    volumes:
      - app-volume:/python-model/server/classify_crack/:rw
    ports:
      - "6379:6379"
    command: ["redis-server"]

  python-model: &python-model
    build:
      context: /Users/AjayB/Desktop/Python/python/
    ports:
      - "8001:8001"
    networks:
      - app-tier
    environment:
      - PYTHON_API_ENV=development
    volumes:
      - app-volume
    depends_on:
      - python-api
    command: >
      sh -c "ls '/python-model/server/classify_crack/inference/images' &&
             cd /python-model/server/ &&
             python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8001"

Instance of containers:

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                    NAMES
7374a7b0b051        integrated_python-celery       "sh -c 'celery -A se…"   13 minutes ago      Up 5 seconds        8000/tcp                 integrated_python-celery_1
8eb9a754996a        integrated_python-model        "sh -c 'cd /python-m…"   20 minutes ago      Up 5 seconds        0.0.0.0:8001->8001/tcp   integrated_python-model_1
b268b7bd1ac4        integrated_python-api-celery   "sh -c 'python manag…"   20 minutes ago      Up 6 seconds        8000/tcp                 integrated_python-api-celery_1
869bb5fc21b2        integrated_python-api          "sh -c 'python manag…"   20 minutes ago      Up 6 seconds        0.0.0.0:8000->8000/tcp   integrated_python-api_1
c85a1becea34        redis:5.0.8-alpine             "docker-entrypoint.s…"   About an hour ago   Up 6 seconds        0.0.0.0:6379->6379/tcp   integrated_redis_1

Answer 1

在我看来，您应该遵循有关如何在 Docker 和 Kubernetes 中使用 celery 和 python API 的深入教程:)

Answer 2

From the OpenCV-Python doc: `

Even if the image path is wrong, it won't throw any error, but print img will give you None

Which seems exactly what happens here. Maybe your image path seems right but is actually wrong, possible causes:

• You are using an incorrect relative or absolute path
• Image file exists but is not valid
• Rights on folder or file /python-model/server/classify_crack/inference/images/202003251237371/202003251237371.jpg does not allow the user running python-celery to read it

Your Compose volume definition looks fine otherwise.

Answer 3

The error got solved at last:

It worked with including following lines in python-api's views.py at the start:

    self.cur_file_dir_path = '/data/'
    self.cur_file_folder_path = '/'.join(file_path.split('/')[:-1])
    if not os.path.exists('{}/inference'.format(self.cur_file_folder_path)):
      os.makedirs('{}/inference'.format(self.cur_file_folder_path))

also, modified the volume name in docker-compose , just to avoid some confusion. Can't believe it took almost 1 week to get this solved.

docker-compose:

version: "3"

networks:
  app-tier:
    driver: bridge

volumes:
  app-volume: {}

services:
  python-api-celery: &python-api-celery
    build:
      context: /Users/AjayB/Desktop/python-api/
    networks:
      - app-tier
    volumes:
      - app-volume:/data/:rw
    environment:
      - PYTHON_API_ENV=development

    command: >
      sh -c "python manage.py makemigrations &&
             python manage.py migrate"

  python-api: &python-api
    <<: *python-api-celery
    ports:
      - "8000:8000"
    command: >
      sh -c "python manage.py runserver 0.0.0.0:8000"

  python-celery: &python-celery
    <<: *python-api-celery
    depends_on:
      - redis
    command: >
      sh -c "ls '/data/' &&
             celery -A server worker -l info"

  redis:
    image: redis:5.0.8-alpine
    hostname: redis
    networks:
          - app-tier
    expose:
      - "6379"
    volumes:
      - app-volume:/data/:rw
    ports:
      - "6379:6379"
    command: ["redis-server"]

  python-model: &python-model
    build:
      context: /Users/AjayB/Desktop/Python/python/
    ports:
      - "8001:8001"
    networks:
      - app-tier
    environment:
      - PYTHON_API_ENV=development
    volumes:
      - app-volume:/data/:rw
    depends_on:
      - python-api

    command: >
      sh -c "ls '/data/' &&
             cd /python-model/server/ &&
             python manage.py migrate &&
             python manage.py runserver 0.0.0.0:8001"