無法從 docker 構建中的工件注冊表安裝私有依賴項

Question

我正在嘗試安裝一個私有的 python package，它已上傳到 docker 容器內的工件注冊表（以將其部署在 cloudrun 上）。

我過去在雲 function 中成功地使用了 package，所以我確信 package 有效。

cloudbuild.yaml

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']

Dockerfile

FROM python:3.8.6-slim-buster

ENV APP_PATH=/usr/src/app
ENV PORT=8080

# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython 

RUN pip install --upgrade pip

# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH

COPY / $APP_PATH

WORKDIR $APP_PATH

RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs


# Expose port 
EXPOSE $PORT

# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app

我添加的權限是：

• cloudbuild 默認服務帳戶(project-number@cloudbuild.gserviceaccount.com)： Artifact Registry Reader
• 運行 cloudbuild 的服務帳戶： Artifact Registry Reader
• 運行應用程序的服務帳戶： Artifact Registry Reader

雲構建錯誤：

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__

Answer 1

從您的回溯日志中，我們可以看到 Cloud Build 沒有向私有存儲庫進行身份驗證的憑據：

Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME

我將一個簡單的 package 上傳到私有 Artifact Registry 存儲庫以在構建容器時對此進行測試，並且也收到了相同的消息。 由於您似乎正在使用服務帳戶密鑰進行身份驗證，因此用戶名和密碼需要存儲在pip.conf ：

pip.conf

[global]
extra-index-url = https://_json_key_base64:KEY@LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/

因此，該文件需要在構建過程中可用。 多階段docker 構建在這里非常有用，可確保配置密鑰不會暴露，因為我們可以選擇將哪些文件放入最終映像（配置密鑰僅在用於從私有倉庫下載包時出現）：

樣品 Dockerfile

# Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build

# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf

WORKDIR /packages/
COPY requirements.txt /

# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf

# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES

# Final image that will be deployed
FROM python:3.8.6-slim-buster

ENV PYTHONUNBUFFERED True
ENV APP_HOME /app

WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/

# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*

WORKDIR $APP_HOME
COPY ./src/main.py ./

# Executing sample flask web app 
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

我基於上面的 dockerfile 這個相關線程，我可以確認這些包是從我的私人 Artifact Registry 存儲庫正確下載的，而且pip.conf文件不存在於結果圖像中。