apt-get update fails on Ubuntu 22 base docker image

Question

I am trying to upgrade one of our docker base images to the latest stable version of Ubuntu. I have isolated the problem to a simple reproducible case. I have a Dockerfile like this:

FROM ubuntu:22.04

MAINTAINER mep-dev@zulily.com

# Install java and clean-up
RUN apt-get update

When I build it on my local machine, I don't have any problems. However, when I build it on my CICD, I sometimes get this error:

Step 3/3: RUN apt-get update
---> Running in 6ca01b60de64
Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Reading package lists...
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease : The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/ .deb /var/cache/apt/archives/partial/ .deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
The command '/bin/sh -c apt-get update' returned a non-zero code: 100

If I comment out RUN apt-get update , then it succeeds, and I can enter the container and see that /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg and /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg do exist and have read for all permissions:

root@b778220b39d8:/# ls -l /etc/apt/trusted.gpg.d
total 8
-rw-r--r-- 1 root root 2794 Mar 26 2021 ubuntu-keyring-2012-cdimage.gpg
-rw-r--r-- 1 root root 1733 Mar 26 2021 ubuntu-keyring-2018-archive.gpg

I also checked the parent directories, and they have rx at least for all.

This answer might be related, but why would the file have the correct structure when the base image is used in one environment and not another?

Update:

By using --pull , I can see the exact image it's using.

$ docker build --pull -t $EMAIL_DELIVERABILITY_ARN.
Step 1/3: FROM ubuntu:22.04
22.04: Pulling from library/ubuntu
Digest: sha256:b6b83d3c331794420340093eb706a6f152d9c1fa51b262d9bf34594887c2c7ac
Status: Downloaded newer image for ubuntu:22.04
---> 27941809078c

This is the same sha and image id that I see when building locally, which works.

I am not having the same problem with ubuntu:20.04.

Answer 1

Exactly same problem

Err:3 http://security.ubuntu.com/ubuntu jammy-security InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C Reading package lists... [91mW: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.

Dockerfile (simple how to reproduce) docker-version: 18.x or 19.03/

FROM ubuntu:latest RUN apt-get -y update

With latest Docker version 20.10.9, i did not see the issue.

some of the options tried to tshoot:

1. sed -i 's/jammy/focal/g' /etc/apt/sources.list
2. RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 871920D1991BC93C this fails with with another child-issue gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

SO, solutions seems to be either update docker-version or use a tagged image where it has worked previously.

Answer 2

When you reference a docker image by name, Docker only checks to see if it exists locally or not -- it doesn't check for updates. So if there is already a version of the named image available, it will use that even though it might be stale. I suspect that's what you're seeing: some of your CI nodes must have a cached version of the image that has problems.

There are a few ways of dealing with this:

1. Explicitly docker pull ubuntu:22.04 before calling docker run ; this will pull down a newer version of the image if one exists.

2. Add --pull always to your docker run command line. This accomplishes the same thing but without requiring an extra command execution.

3. Reference an image by digest rather than by tag. If you use an image reference like this:

    FROM ubuntu@sha256:bace9fb0d5923a675c894d5c815da75ffe35e24970166a48a4460a48ae6e0d19

Then docker will use that exact image . You can find the image digests on docker hub.

Answer 3

It is the same from me, something is broken inside the Ubuntu image. The issue is not coming from the docker software/pkg, it is sourced within the image.

Answer 4

After spending half a day trying to fix the Ubuntu images (which aren't broken) I have eventually started debugging the host.

It's a docker problem. Ubuntu makes use of syscalls for better key security, which Docker didn't support yet. The solution is to update docker... or use runc or something similar.

Instead of apt getting the message that the syscalls aren't supported, it gets the message that permission is denied, which results in the confusing error messages.

You could technically patch ubuntu to be less secure, and to work with older docker, but that is sadly not a long term solution.