Debug failed Docker builds on Gitlab CI and execute intermediate layers

Question

I wonder how it would be possible to debug a Docker build by executing an intermediate build layer and run a debug container out of the layer to watch what is inside.

Because I found no answer anywhere, I created my custom solution, which works pretty well (see below).

Answer 1

Solution

I added a debug-failed-build job to my pipeline, which uploads the layer as docker image to a Gitlabs Docker registry:

.gitlab-registry-login: &local-registry-login
  docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

build:
  stage: build
  script:
    - *local-registry-login
    - docker build --pull -t "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}" . | tee docker-build-debug.out
    - docker push "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}"
  artifacts:
    paths:
      - docker-build-debug.out
    when: on_failure
    expire_in: 30 mins

debug-failed-build:
  stage: debug
  script:
    - *local-registry-login
    - DEBUG_LAYER=$(grep '\-\-\-> [0-9a-z]' docker-build-debug.out |tail -1| cut -b 7-)
    - docker tag "$DEBUG_LAYER" "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}-failed"
    - docker push "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_SLUG}-failed"
  when: on_failure
  dependencies:
    - build

How it works

The output of the Docker build is stored in a file which in case of failure is passed as an artifact to the debug-failed-build job. Here is an example how the output of a Docker build could look like (just a snippet):

Step 16/19 : VOLUME ["/sys/fs/cgroup"]
 ---> Using cache
 ---> a63a68682fcb
Step 17/19 : COPY --from=ansibleci-base /ansibleci-base /ansibleci-base
 ---> Using cache
 ---> 98fa646b73fb
Step 18/19 : RUN ln -s /ansibleci-base/scripts/run-tests.sh /usr/local/bin/run-tests &&     ln -s /ansibleci-base/ansible-plugins/human_log.py /usr/local/lib/python3.6/dist-packages/ansible/plugins/callback/human_log.py
 ---> Running in 83116392053c
ln: failed to create symbolic link '/usr/local/lib/python3.6/dist-packages/ansible/plugins/callback/human_log.py': No such file or directory

The expression behind the DEBUG_LAYER=... script command will extract the last layer id from the Docker build output ( 98fa646b73fb ). The next command will give this layer an image name ready to upload to the registry and the final command will upload that image.

As an alternative to uploading the image you can also save the layer as file (with docker save ) and store the saved image as compressed tar archive. Then you define this archife as Gitlab CI Artifact which you can download to your computer and docker load it there.