Docker image pull fails with not found: manifest

Question

I'm using Docker version 20.10.8, build 3967b7d on CentOS-7.9 host and Sonatype Nexus Repository Manager version - OSS 3.30.1-01 being used as private docker registry. When i tried to pull the image using docker-compose it always fails with below error.

$ docker-compose up -d
Pulling prometheus (internal-registry.com:8335/prometheus:latest)...
ERROR: manifest for internal-registry.com:8335/prometheus:latest not found: manifest unknown: manifest unknown

Error from Sonatype Nexus server log as follows.

2021-10-11 10:35:41,307+0530 WARN  [qtp657241891-125]  admin org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: HEAD /v2/library/prometheus/manifests/latest: 401 - org.sonatype.nexus.repository.docker.internal.V2Exception: authentication required
2021-10-11 10:35:42,106+0530 WARN  [qtp657241891-560]  admin org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: GET /v2/library/prometheus/manifests/latest: 401 - org.sonatype.nexus.repository.docker.internal.V2Exception: authentication required.

Docker host docker login to our private registry is successful.

1) Repo - Allow clients to use the V1 API to interact with this repository (Tick mark enabled).
2) DockerHub(Proxy) - Allow clients to use the V1 API to interact with this repository (Tick mark enabled).
3) Group - Allow clients to use the V1 API to interact with this repository (Tick mark enabled).

4) Nexus UI > Security > Anonymous Access > Allow anonymous users to access the server - (Tick mark enabled).
Username: anonymous
Realm: Local Autherizing Realm

5) Nexus UI > Security > Role > Create role > Create Nexus role > Created new role
Privileges: nx-repository-view-*-*-*-broswe
            nx-repository-view-*-*-*-read
Roles: Contained - nx-anonymous

6) Nexus UI > Security > Users > anonymous(Active)
         Roles: Granted: read_and-browse

7) Nexus UI > Security > Realms - Docker Bearer Token Realm (Is in Active List) 

Still getting not found: manifest unknown: manifest unknown error while pulling docker image. How do i solve this problem? Any pointer would be helpful. Thanks!

Answer 1

This looks like some kind of network/memory/hardware issue, probably on internal-registry.com, assuming that prometheus is an available image on that registry. I would first load test that service, then verify dns, then the firewall, then the memory, then the power supply, then the hardware of that service to troubleshoot it.

Finally, I would try to recreate this error in another environment, again, against internal-registry.com. If you can't, then the issue is with the docker client, but I suspect that you will be able to replicate this issue on other clients.

Answer 2

1. Make sure the docker image (Above was wrong).

2. This Link has helped to create a Read & Browse privilege on Repository in the Nuxus side. Post that able to pull the image successfully.