Gitlab docker注册表拉失EOF

Question

1. Introduction

I've got Gitlab setup using a docker image https://github.com/sameersbn/docker-gitlab (sameersbn/docker-gitlab) and want to setup gitlab private registry.

1.1 Problem

All of this is behind reverse proxy load balancer using nginx . I am able to push into registry, but pull has some problem I am not able to resolve.

The output tells me retrying in 5sec and then increases over time, but after about minute it outputs Download complete on some layers. When all layers outputs Download complete . Pull results then like Unexpected EOF .

Infrastructure goes like this:

Proxy load balancer -----> gitlab registry | | Another proxy load balancer -----> Gitlab

2. Terminal outputs

Docker login and push works fine. The problem is with pull.

2.1 Docker pull

2.1.1 expected result

root@server:~# docker pull registry.dalten.com/docker/php5.6-cli/image
Using default tag: latest
latest: Pulling from docker/php5.6-cli/image
10a267c67f42: Already exists 
370377701f89: Pulling fs layer 
455c73a122bc: Pulling fs layer 
2cdcd82f5338: Pulling fs layer 
3ac634f20449: Waiting 
a19eeb6546ae: Waiting 
4ca536f5a464: Waiting 
4788559f39db: Waiting 
cc1917e72a80: Waiting 
42feb0ca362a: Pull complete 
f74f29b91bbc: Pull complete 
d104213bfd22: Pull complete 
6db9d2d35d48: Pull complete 
806b8ac2f0b7: Pull complete 
661bb76178ac: Pull complete 
3727be25d45f: Pull complete 
04fed213644a: Pull complete 
a78f18e044f7: Pull complete 
849a2f576c66: Pull complete 
259300decf26: Pull complete 
0d9f42b121be: Pull complete 
2483d273017a: Pull complete 
3dfaf57e1d54: Pull complete 
92fbf8438206: Pull complete 
7d5100ec575c: Pull complete 
52b1dff7320d: Pull complete 
500fa7f84392: Pull complete 
aff937c2773b: Pull complete 
92f8bc1201f3: Pull complete 
Digest: sha256:d3d06d2cebd333af21565639503ddded64db47d5641ee9ff93167200b5fcad7a
Status: Downloaded newer image for registry.dalten.com/docker/php5.6-cli/image:latest

2.1.2 Actual behavior

docker pull registry.dalten.com/docker/php5.6-cli/image
Using default tag: latest
latest: Pulling from docker/php5.6-cli/image
10a267c67f42: Already exists 
370377701f89: Pulling fs layer 
455c73a122bc: Pulling fs layer 
2cdcd82f5338: Pulling fs layer 
3ac634f20449: Waiting 
a19eeb6546ae: Waiting 
a19eeb6546ae: Downloading 4.398 MB/4.398 MB
4788559f39db: Downloading 102.6 MB/102.6 MB
cc1917e72a80: Download complete 
42feb0ca362a: Download complete 
f74f29b91bbc: Download complete 
d104213bfd22: Download complete 
6db9d2d35d48: Download complete 
806b8ac2f0b7: Download complete 
661bb76178ac: Download complete 
3727be25d45f: Download complete 
04fed213644a: Download complete 
a78f18e044f7: Download complete 
849a2f576c66: Download complete 
259300decf26: Download complete 
0d9f42b121be: Download complete 
2483d273017a: Downloading 24.12 MB/24.12 MB
3dfaf57e1d54: Download complete 
92fbf8438206: Download complete 
7d5100ec575c: Download complete 
52b1dff7320d: Download complete 
500fa7f84392: Download complete 
aff937c2773b: Download complete 
92f8bc1201f3: Download complete 
unexpected EOF

2.3 Docker login

sudo docker login registry.dalten.com
Username (xxxxxx): 
Password: 
Login Succeeded

2.4 Docker push

sudo docker push registry.dalten.com/docker/php5.6-cli/image
The push refers to a repository [registry.dalten.com/docker/php5.6-cli/image]
d8b353eb3025: Pushed 
f2e85bc0b7b1: Pushed 
fc9e1e5e38f7: Pushed 
fe9a3f9c4559: Pushed 
6a8bf8c8edbd: Pushed 
latest: digest: sha256:ea1d854d38be82f54d39efe2c67000bed1b03348bcc2f3dc094f260855dff368 size: 1357

3. Configurations

Configurations contains docker-compose.yml files which I use for running services. Gitlab is running fine. Every configuration works. The only problem is with registry.

Another thing is that registry works while in insecured state. The problem must be hidden somewhere inside load balancer (nginx) configuration. (Some header has not been sent, limit exceeded, or so)

3.1 Gitlab Docker-compose

Note: I cutted compose of irrelevant information

version: '2'
services:
  registry:
    image: registry:latest
    environment:
      REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
      REGISTRY_AUTH_TOKEN_REALM: https://git.dalten.cz/jwt/auth
      REGISTRY_AUTH_TOKEN_SERVICE: container_registry
      REGISTRY_LOG_LEVEL: debug
      REGISTRY_STORAGE_DELETE_ENABLED: 'true'
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/fullchain.pem
      REGISTRY_HTTP_SECRET: secret
    volumes:
    - /mnt/storage/srv/gitlab1/data/shared/registry:/registry
    - /mnt/storage/srv/gitlab1/certs:/certs
    ports:
    - 5000:5000/tcp
  gitlab:
    image: sameersbn/gitlab:latest
    hostname: git.dalten.cz
    environment:
      GITLAB_REGISTRY_ENABLED: 'true'
      GITLAB_REGISTRY_HOST: registry.dalten.com
      GITLAB_REGISTRY_API_URL: http://registry:5000
      GITLAB_REGISTRY_ISSUER: gitlab-issuer
      GITLAB_REGISTRY_CERT_PATH: /certs/fullchain.pem
      GITLAB_REGISTRY_KEY_PATH: /certs/privkey.pem
    volumes:
    - /mnt/storage/srv/gitlab1/certs:/certs
    - /mnt/storage/srv/gitlab1/data:/home/git/data/
    ports:
    - 8643:80/tcp
    - 1022:22/tcp

3.2 Nginx registry docker-compose

version: '2'
services:
  nginx:
    image: sameersbn/nginx:1.10.3
    volumes:
    - /srv/nginx/sites-enabled:/etc/nginx/sites-enabled
    - /mnt/storage/lb/letsencrypt:/etc/nginx/ssl
    ports:
    - 80:80/tcp
    - 443:443/tcp

3.2.1 Nginx Virtual host file

I use Nginx for my reverse proxy configuration (Load Balancer). There are stored all certificates for https traffic.

    # For versions of Nginx > 1.3.9 that include chunked transfer encoding
# support Replace with appropriate values where necessary
upstream docker-registry {
 server 10.126.0.242:5000;
}

server {
  listen 80;
  server_name registry.dalten.com
  return 301 https://$http_host:$request_uri;
  access_log /var/log/nginx/access.log;
  error_log  /var/log/nginx/error.log;
}

server {
 listen 443 default_server;
 server_name registry.dalten.com

 add_header Docker-Distribution-Api-Version registry/2.0 always;

 ssl on;
 ssl_certificate /etc/nginx/ssl/live/registry.dalten.com/fullchain.pem;
 ssl_certificate_key /etc/nginx/ssl/live/registry.dalten.com/privkey.pem;

  ssl_session_timeout 5m;

 client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image
uploads
 # required to avoid HTTP 411: see Issue #1486
 # (https://github.com/dotcloud/docker/issues/1486)
 chunked_transfer_encoding on;
 proxy_set_header X-Original-URI $request_uri;
 proxy_set_header Docker-Distribution-Api-Version registry/2.0;

access_log /var/log/nginx/access.log;
 error_log /var/log/nginx/error.log;

location /
{
     # let Nginx know about our auth file
     proxy_pass http://docker-registry;
     proxy_set_header Host $host; # required for docker client's sake
     proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto $scheme;
 }
 location /_ping {
     proxy_pass http://docker-registry;
 }
 location /v1/_ping {
     proxy_pass http://docker-registry;
 }
 location /v2/ {
     # To add basic authentication to v2 use auth_basic setting plus
     # add_header
     add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
     proxy_pass http://docker-registry;
     proxy_set_header Host $http_host; # required for docker client's sake
     proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto $scheme;
     proxy_read_timeout 900;
 }
}

3. Logs

If you require any more logs feel free to ask!

3.1 Docker pull

4. Thank you for all your help!

Answer 1

Problem solved by adding proxy_max_temp_file_size 0; inside nginx virtual host configuration!