gitlab CE docker push to registry hosted in s3 fails with “denied: access forbidden”

Question

In gitlab CE docker push to registry hosted in s3 fails with "denied: access forbidden"

Here is a sample of my gitlab-ci.yml file:

    before_script:
      - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN gitlab.domain.net (successful)

    Build:
      stage: build
      script:
        - docker build --pull -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME .  (successful)
        - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME    (FAILS)

Result: Fails during the push.

    The push refers to repository [gitlab.domain.net:6565/sfloyd/myapp]
    7f44c19d438d: Preparing
    26472437084c: Preparing
    2de391e51d73: Preparing
    d73dd9e65295: Preparing
    686245e78935: Preparing
    d7ff1dc646ba: Preparing
    644879075e24: Preparing
    d7ff1dc646ba: Waiting
    644879075e24: Waiting
    denied: access forbidden
    ERROR: Job failed: exit code 1

I have tried not using S3 and having the registry store the images on the gitlab server itself and that fails with the same result. I even set the S3 bucket open to the world for read and write as a test and that also failed. The CI_JOB_TOKEN was given "read_registry" access just like the Gitlab documentation states...although I feel like it should have write as well but its not an option. I have swapped out the username to log in with, with the username of the token and it fails with the same result. I created an access token for my admin user giving it API access and that fails. Interestingly if i use my admin user and the access token with API access it works LOCALLY but not when I put those same values in the gitlab-ci.yml file.

    $ docker images
    REPOSITORY                            TAG                 IMAGE ID                            CREATED             SIZE
    gitlab.domain.net:6565/sfloyd/myapp   master              b03b47a5987d        14 seconds ago      211MB
    ubuntu                                16.04                       5e8b97a2a082        2 days ago          114MB
    $ docker push gitlab.domain.net:6565/sfloyd/myapp:master
    The push refers to repository [gitlab.domain.net:6565/sfloyd/myapp]
    6852ea579a31: Pushed
    5f7a44809386: Pushed
    2de391e51d73: Layer already exists
    d73dd9e65295: Layer already exists
    686245e78935: Layer already exists
    d7ff1dc646ba: Layer already exists
    644879075e24: Layer already exists
    master: digest:         sha256:0f9f6c3d6ebc7f23f52a5b7be06457359de1c24959336de5b5e9e0a208a5b8c8         size: 1781

Ive pretty much exhausted everything I can think of. Anyone have a suggestion?

GitLab 10.8.1 Gitlab Runner 10.8.0 Also not sure if this matters but this is a private project.

Answer 1

Got it working in case someone else runs into this...

For some reason this worked as my login: docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY

But did not work when I hardcoded those values in.