Docker - 無法使用 docker buildx 構建多平台映像

Question

我正在嘗試使用 docker buildx 構建多平台（amd64、arm64 和 armv7）映像。 由於我使用的是運行 Ubuntu 18.04 的 amd64 機器，因此我按照Docker 網站上的說明進行操作，並通過以下方式安裝了qemu ：

sudo apt install qemu-user

但是，當我執行上一個命令時出現了一個奇怪的錯誤。 更具體地說， binfmt-support服務似乎存在問題。 這是完整的日志：

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
  binfmt-support qemu-user-binfmt
The following NEW packages will be installed:
  binfmt-support qemu-user qemu-user-binfmt
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/7.409 kB of archives.
After this operation, 63,4 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Selecting previously unselected package binfmt-support.
(Reading database ... 245278 files and directories currently installed.)
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package qemu-user.
Preparing to unpack .../qemu-user_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Selecting previously unselected package qemu-user-binfmt.
Preparing to unpack .../qemu-user-binfmt_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Setting up binfmt-support (2.1.8-2) ...
Job for binfmt-support.service failed because the control process exited with error code.
See "systemctl status binfmt-support.service" and "journalctl -xe" for details.
invoke-rc.d: initscript binfmt-support, action "start" failed.
● binfmt-support.service - Enable support for additional executable binary formats
   Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-02-05 17:20:29 CET; 4ms ago
     Docs: man:update-binfmts(8)
  Process: 7766 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, status=2)
 Main PID: 7766 (code=exited, status=2)

feb 05 17:20:29 XPS-15-9570 systemd[1]: Starting Enable support for additional executable binary formats...
feb 05 17:20:29 XPS-15-9570 update-binfmts[7766]: update-binfmts: warning: unable to close /proc/sys/fs/binfmt_misc/register: No such file or directory
feb 05 17:20:29 XPS-15-9570 update-binfmts[7766]: update-binfmts: exiting due to previous errors
feb 05 17:20:29 XPS-15-9570 systemd[1]: binfmt-support.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
feb 05 17:20:29 XPS-15-9570 systemd[1]: binfmt-support.service: Failed with result 'exit-code'.
feb 05 17:20:29 XPS-15-9570 systemd[1]: Failed to start Enable support for additional executable binary formats.
Setting up qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Setting up qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
update-binfmts: warning: current package is qemu-user-binfmt, but binary format already installed by qemu-user-static
update-binfmts: exiting due to previous errors
dpkg: error processing package qemu-user-binfmt (--configure):
 installed qemu-user-binfmt package post-installation script subprocess returned error exit status 2
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for neon-settings (0.0+p18.04+git20191212.1343) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Errors were encountered while processing:
 qemu-user-binfmt
E: Sub-process /usr/bin/dpkg returned an error code (1)

盡管如此，我還是嘗試使用通常的程序打開 go，即：

docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap

其中最后一條命令的 output 為：

[+] Building 5.0s (1/1) FINISHED                                                                                                                                                             
 => [internal] booting buildkit                                                                                                                                                         5.0s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                      4.3s
 => => creating container buildx_buildkit_mybuilder0                                                                                                                                    0.7s
Name:   mybuilder
Driver: docker-container

Nodes:
Name:      mybuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/386

如您所見，“linux/amd64”和“linux/386”被列為唯一可用的平台，但是我還需要為“linux/arm64”和“linux/arm/v7”平台構建映像。 幾個小時以來，我一直在尋找解決這個問題的方法，但我沒有找到任何有效的方法

---

- - - - - - - - - - - - - - - - - - 編輯 - - - - - - - ----------------------

看起來我能夠通過運行解決部分問題：

sudo apt purge --auto-remove qemu-user qemu-user-binfmt binfmt-support

然后重新安裝它們。 事實上，再次運行這個命令：

sudo apt install qemu-user

完全沒有錯誤：

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
The following additional packages will be installed:
  binfmt-support qemu-user-binfmt
The following NEW packages will be installed:
  binfmt-support qemu-user qemu-user-binfmt
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/7.409 kB of archives.
After this operation, 63,4 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Selecting previously unselected package binfmt-support.
(Reading database ... 245437 files and directories currently installed.)
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Selecting previously unselected package qemu-user.
Preparing to unpack .../qemu-user_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Selecting previously unselected package qemu-user-binfmt.
Preparing to unpack .../qemu-user-binfmt_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...
Unpacking qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Setting up binfmt-support (2.1.8-2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/binfmt-support.service → /lib/systemd/system/binfmt-support.service.
Setting up qemu-user (1:2.11+dfsg-1ubuntu7.21) ...
Setting up qemu-user-binfmt (1:2.11+dfsg-1ubuntu7.21) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for neon-settings (0.0+p18.04+git20191212.1343) ...
Processing triggers for systemd (237-3ubuntu10.38) ...

同樣， systemctl status binfmt-support.service的 output 也符合預期：

● binfmt-support.service - Enable support for additional executable binary formats
   Loaded: loaded (/lib/systemd/system/binfmt-support.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2020-02-10 11:42:23 CET; 1min 11s ago
     Docs: man:update-binfmts(8)
 Main PID: 7161 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/binfmt-support.service

feb 10 11:42:23 XPS-15-9570 systemd[1]: Starting Enable support for additional executable binary formats...
feb 10 11:42:23 XPS-15-9570 systemd[1]: Started Enable support for additional executable binary formats.

但是，部分問題仍然存在，因為運行這三個命令后出現 output：

docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx inspect --bootstrap

與之前相同，即：

[+] Building 2.6s (1/1) FINISHED                                                                                                                                                             
 => [internal] booting buildkit                                                                                                                                                         2.6s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                                      2.0s
 => => creating container buildx_buildkit_mybuilder0                                                                                                                                    0.6s
Name:   mybuilder
Driver: docker-container

Nodes:
Name:      mybuilder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/386

這是為什么？ 為什么它顯示linux/amd64和linux/386是唯一可用的平台？

---

編輯#2（關於@LinPy 的評論）

docker context ls的output是：

NAME                DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                         swarm

我也試過在安裝 qemu 后重啟 docker，但沒有成功。 此外，在 docker buildx 命令中指定目標平台：

docker buildx build -t <mytag> --platform linux/amd64,linux/arm64,linux/arm/v7 --load .

導致此錯誤：

[+] Building 0.6s (5/20)                                                                                                      
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 2B                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 32B                                                                                      0.0s
 => [linux/arm/v7 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2                                   0.6s
 => CANCELED [linux/arm64 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2                           0.6s
 => CANCELED [linux/amd64 internal] load metadata for docker.io/alegeno92/opencv_python3:3.4.2                           0.6s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to load LLB: runtime execution on platform linux/arm/v7 not supported

對了，我的kernel版本是4.15.0-76-generic

Answer 1

首先運行多架構容器

 docker run --rm --privileged multiarch/qemu-user-static --reset -p yes docker buildx rm builder docker buildx create --name builder --driver docker-container --use docker buildx inspect --bootstrap

你應該有你的替代架構。

Answer 2

有多個 binfmt 包，我認為在問這個問題時遺漏了一個配置。

對於各種包，我會選擇qemu-user-static而不是qemu-user-binfmt以避免任何動態鏈接問題。 這兩個包做同樣的事情，所以你需要選擇一個。

下一部分應該在當前版本中得到修復，但我認為您之前在這方面遇到過困難。 那是你在/proc/sys/fs/binfmt_misc中捕獲文件時會看到的fix binary或F標志，例如，請在此處查看F標志：

$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/libexec/qemu-binfmt/arm-binfmt-P
flags: POCF
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

有關F標志含義的詳細信息，請參閱此 kernel.org 帖子，但它的缺點是容器名稱空間包含不同的文件系統名稱空間，並且嘗試從該名稱空間訪問解釋器將失敗（除非您執行類似 bind mount /usr/libexec/qemu-binfmt到你的容器中）。 較新版本的 qemu 軟件包會自動設置此標志，因此如果您的標志部分沒有定義F ，請參閱這些錯誤報告以了解您需要升級到的版本：

• Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868030
• Ubuntu: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1815100

---

簡單的按鈕是使用多體系結構映像中的二進制文件。 如果您有專用的 VM，這在 CI 中很好（如果您正在修改其他構建使用的主機，則不太理想）。 但是，如果您重新啟動，它將中斷，直到您再次運行該容器。 它要求您記住為任何上游補丁更新它。 所以我不會推薦它用於長時間運行的構建主機。

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Answer 3

標記此答案以響應第一個錯誤。 這些命令已根據https://docs.docker.com/buildx/working-with-buildx/進行更新。

QEMU 是一個跨平台模擬器，負責為不同的架構（通過 binfmt_misc 處理程序）獲取二進制文件。

將節省一些人一些時間來首先使用此命令：

docker run --privileged --rm tonistiigi/binfmt --install all

Answer 4

對於 github CI，添加以下插件為我解決這個問題

    - name: Set up QEMU
      id: qemu
      uses: docker/setup-qemu-action@v1
      with:
        image: tonistiigi/binfmt:latest
        platforms: all