pulling image from private docker registry causes filesystem layer verification failed for digest sha256:

Question

So I have a Kube.netes cluster which is connected to a private docker registry. One of my nodes in Kube.netes keeps failing at pulling an image. So I went into the node and tried to pull the image manually. When I try to pull the image, it fails. I get the following error: filesystem layer verification failed for digest sha256:...

I tried the following post, but no solution in there worked: filesystem layer verification failed for digest

Answer 1

I have a private docker registry, a layer was failing for verification right after it was downloaded, none of the above solutions worked for me.

I did the following, delete the image and specific layer from the private docker registry. Restart the private docker registry. Then rebuild and push the image again. It should work afterwards.

To remove a tag from a private docker registry I use the following script (Python):

image = "yourimage"
tag = "yourtag"
host = "https://yourhost:5000" # change to http if you have no ssl
username = "yourusername"
password = "yourpassword"
res = requests.get("{}/v2/{}/manifests/{}".format(host, image, tag), auth=HTTPBasicAuth(username, password), verify=False, headers={
        "Accept": "application/vnd.docker.distribution.manifest.v2+json"
    })
digest = res.headers.get("Docker-Content-Digest").replace('"', "")
res = requests.delete("{}/v2/{}/manifests/{}".format(host, image, digest), auth=HTTPBasicAuth(username, password), verify=False)
    print(res.status_code)

to remove the blob, your blob digest is the sha256:.... that fails at verification

image = "yourimage"
host = "https://yourhost:5000" # change to http if you have no ssl
username = "yourusername"
password = "yourpassword"
blob_digest = "yourblobdigest"
res = requests.delete("{}/v2/{}/blobs/{}".format(host, image, blob_digest), auth=HTTPBasicAuth(username, password),
                      verify=False)

after you have finished this go to your container of your private docker registry and remove the image like this:

rm /var/lib/registry/docker/registry/v2/repositories/your-repository-name

Exit your docker registry and restart your private docker registry container. Build and push the image and it should all go fine when when you pull again.