Traefik Proxy 后面的 Mailcow 设置导致 https 证书错误

Question

I am trying to setup the mailcow installation behind Traefik proxy. Apparently, Traefik proxy is not able to recognize the nginx-mailcow container in its network and hence does not create a certificate for https connection. so when I bring up the mailcow service using docker-compose up , I can access the mailcow services but on insecure connection (http) and browser warns that connection is not secure.

When I check my acme.json file from Traefik : I can not find any certificate related to mailcow domain ie, mail.tld.com there.

I have the following setup:

Logs of affected containers:

Traefik Container Logs:

time="2020-04-18T13:40:35+02:00" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=http
time="2020-04-18T13:40:35+02:00" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=https
time="2020-04-18T13:40:35+02:00" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=http
time="2020-04-18T13:40:35+02:00" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=https
time="2020-04-18T13:40:35+02:00" level=error msg="Cannot connect to docker server context canceled" providerName=docker
time="2020-04-18T13:40:37+02:00" level=info msg="Configuration loaded from file: /traefik.yml"
time="2020-04-19T00:27:31+02:00" level=error msg="service \"nginx-mailcow\" error: unable to find the IP address for the container \"/mailcowdockerized_nginx-mailcow_1\": the server is ignored" container=nginx-mailcow-mailcowdockerized-5f3a25b43c42fd85df675d2d9682b6053501844c2cfe15b7802cf918df138025 providerName=docker
time="2020-04-19T00:33:32+02:00" level=error msg="service \"nginx-mailcow\" error: unable to find the IP address for the container \"/mailcowdockerized_nginx-mailcow_1\": the server is ignored" providerName=docker container=nginx-mailcow-mailcowdockerized-f4d41ee79e382b413e04b039b5fc91e1c6217c78740245c8666373fe2d6a9b23
2020/04/19 00:39:44 reverseproxy.go:445: httputil: ReverseProxy read error during body copy: unexpected EOF
time="2020-04-19T00:50:32+02:00" level=error msg="service \"nginx-mailcow\" error: unable to find the IP address for the container \"/mailcowdockerized_nginx-mailcow_1\": the server is ignored" providerName=docker container=nginx-mailcow-mailcowdockerized-915f80e492c2c22917d0af81add1dde15577173c82cc928b0b6101c8a260adc5
time="2020-04-19T00:58:43+02:00" level=error msg="service \"nginx-mailcow\" error: unable to find the IP address for the container \"/mailcowdockerized_nginx-mailcow_1\": the server is ignored" container=nginx-mailcow-mailcowdockerized-852985c4efc48559ca3568b1829e31b46eb9f968fc328a8566e3dc6ab6f1af21 providerName=docker
time="2020-04-19T02:02:39+02:00" level=error msg="Error while Peeking first byte: read tcp 172.21.0.2:80->208.91.109.90:55153: read: connection reset by peer"
time="2020-04-19T08:11:32+02:00" level=error msg="service \"nginx-mailcow\" error: unable to find the IP address for the container \"/mailcowdockerized_nginx-mailcow_1\": the server is ignored" providerName=docker container=nginx-mailcow-mailcowdockerized-840ef4db0ccc9fa84038dc7a52133779926dba4c51554516c17404ede80a2c01

The contents of Traefik docker-compose.yml :

version: '3'

services:
  traefik:
    image: traefik:v2.1
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/traefik.yml:/traefik.yml:ro
      - ./data/acme.json:/acme.json
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.tld.com`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=admin:pass"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.tld.com`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=http"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

Contents of traefik.yml (I used.yml instead of.toml)

api:
    dashboard: true

entryPoints:
    http:
        address: ":80"
    https:
        address: ":443"

providers:
    docker:
        endpoint: "unix:///var/run/docker.sock"
        exposedByDefault: false

certificatesResolvers:
    http:
        acme:
            email: myemail@tld.com
            storage: acme.json
            httpChallenge:
                entryPoint: http

Just to point out, with this setup of Traefik , certificates are generated automatically for other services like gitlab . For that, I just correctly labelled the gitlab service and assigned the Traefik network to it and Traefik service would recognize the gitlab service and generates the certificate in acme.json but sadly not for nginx-mailcow .

The contents of my docker-compose.override.yml for mailcow:

version: '2.1'

services:
  nginx-mailcow:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nginx-mailcow.entrypoints=http"
      - "traefik.http.routers.nginx-mailcow.rule=HostRegexp(`{host:(autodiscover|autoconfig|webmail|mail|email).+}`)"
      - "traefik.http.middlewares.nginx-mailcow-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.nginx-mailcow.middlewares=nginx-mailcow-https-redirect"
      - "traefik.http.routers.nginx-mailcow-secure.entrypoints=https"
      - "traefik.http.routers.nginx-mailcow-secure.rule=Host(`mail.tld.com`)" 
      - "traefik.http.routers.nginx-mailcow-secure.tls=true"
      - "traefik.http.routers.nginx-mailcow-secure.service=nginx-mailcow"
      - "traefik.http.services.nginx-mailcow.loadbalancer.server.port=80"
      - "traefik.docker.network=proxy"
    networks:
      proxy:

  certdumper:
      image: humenius/traefik-certs-dumper
      container_name: traefik_certdumper
      network_mode: none
      command: --restart-containers mailcowdockerized_postfix-mailcow_1,mailcowdockerized_dovecot-mailcow_1
      volumes:
        - /opt/containers/traefik/data:/traefik:ro
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - ./data/assets/ssl:/output:rw
      environment:
        - DOMAIN=tld.com 

networks:
  proxy:
    external: true

The contents of my nginx-mailcow service in docker-compose.yml

version: '2.1'
services:
     ...
    nginx-mailcow:
      depends_on:
        - sogo-mailcow
        - php-fpm-mailcow
        - redis-mailcow
      image: nginx:mainline-alpine
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
      command: /bin/sh -c "envsubst < /etc/nginx/conf.d/templates/listen_plain.template > /etc/nginx/conf.d/listen_plain.active &&
        envsubst < /etc/nginx/conf.d/templates/listen_ssl.template > /etc/nginx/conf.d/listen_ssl.active &&
        envsubst < /etc/nginx/conf.d/templates/server_name.template > /etc/nginx/conf.d/server_name.active &&
        envsubst < /etc/nginx/conf.d/templates/sogo.template > /etc/nginx/conf.d/sogo.active &&
        envsubst < /etc/nginx/conf.d/templates/sogo_eas.template > /etc/nginx/conf.d/sogo_eas.active &&
        . /etc/nginx/conf.d/templates/sogo.auth_request.template.sh > /etc/nginx/conf.d/sogo_proxy_auth.active &&
        . /etc/nginx/conf.d/templates/sites.template.sh > /etc/nginx/conf.d/sites.active &&
        nginx -qt &&
        until ping phpfpm -c1 > /dev/null; do sleep 1; done &&
        until ping sogo -c1 > /dev/null; do sleep 1; done &&
        until ping redis -c1 > /dev/null; do sleep 1; done &&
        until ping rspamd -c1 > /dev/null; do sleep 1; done &&
        exec nginx -g 'daemon off;'"
      environment:
        - HTTPS_PORT=${HTTPS_PORT:-443}
        - HTTP_PORT=${HTTP_PORT:-80}
        - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
        - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1}
        - TZ=${TZ}
        - ALLOW_ADMIN_EMAIL_LOGIN=${ALLOW_ADMIN_EMAIL_LOGIN:-n}
      volumes:
        - ./data/web:/web:ro
        - ./data/conf/rspamd/dynmaps:/dynmaps:ro
        - ./data/assets/ssl/:/etc/ssl/mail/:ro
        - ./data/conf/nginx/:/etc/nginx/conf.d/:rw
        - ./data/conf/rspamd/meta_exporter:/meta_exporter:ro
        - sogo-web-vol-1:/usr/lib/GNUstep/SOGo/
      ports:
        - "${HTTPS_BIND:-0.0.0.0}:${HTTPS_PORT:-443}:${HTTPS_PORT:-443}"
        - "${HTTP_BIND:-0.0.0.0}:${HTTP_PORT:-80}:${HTTP_PORT:-80}"
      restart: always
      networks:
        mailcow-network:
          aliases:
            - nginx

....

I have also tried comment out ports in nginx-mailcow service but the problem persists. My current mailcow.conf changes:

HTTP_BIND=127.0.0.1
HTTP_PORT=8080
HTTPS_BIND=127.0.0.1
HTTPS_PORT=8443

SKIP_LETS_ENCRYPT=y
SKIP_CLAMD=y

Reproduction of said bug:

I setup the traefik proxy first (see contents above). Once the Traefik is up and running (I also tested for other services and it works fine in generating a certificate). Now first I cloned the mailcow repository. Then I run ./generate_config.sh to generate mailcow.conf file. As input to generate_config.sh I provide my domain name ie, mail.tld.com

Then I comment out the ports in docker-compose.yml file because I do not want to use port 80 and 443 for nginx-mailcow as these ports are already being used by Traefik .

Then I create a docker-compose.override.yml (see contents above) to add additional configs to nginx-mailcow service (traefik labels, traefik network). The override file also contain the certdumper service which would copy https certificate from acme.json to mailcow services.

Then, I change the following two variables in mailcow.conf :

SKIP_LETS_ENCRYPT=y
SKIP_CLAMD=y

Finally, I run the mailcow using docker-compose up -d . In browser, if check https://mail.tld.com => It warns that connection is insecure. If I check acme.json . I find no certificate for mail.tld.com .

System information:

+-------------------------------------------------+---------------------------------+
|                    Question                     |             Answer              |
+-------------------------------------------------+---------------------------------+
| My operating system                             | linux x86_64 Ubuntu 18.04.1 LTS |
| Is Apparmor, SELinux or similar active?         | No                              |
| Virtualization technlogy                        | KVM                             |
| Server/VM specifications (Memory, CPU Cores)    | 16GB, 6 cores                   |
| Docker Version (docker version)                 | 19.03.8                         |
| Docker-Compose Version (docker-compose version) | 1.25.4, build 8d51620a          |
| Reverse proxy (custom solution)                 | Traefik                         |
+-------------------------------------------------+---------------------------------+

If you need more information, I would be happy to provide. Any help will be much appreciated. Thank you.

Answer 1

Finally I was able to solve the problem after investing many hours in reading the Traefik Documentation. I made tiny mistake in assigning proxy labels to the nginx-mailcow service. The solution is below.

I forgot to mention certificate resolver and I had to expose the port which I now added as follows:

services:
  nginx-mailcow:
    expose:
      - "8080"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nginx-mailcow.entrypoints=http"
      - "traefik.http.routers.nginx-mailcow.rule=HostRegexp(`{host:(autodiscover|autoconfig|webmail|mail|email).+}`)"
      - "traefik.http.middlewares.nginx-mailcow-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.nginx-mailcow.middlewares=nginx-mailcow-https-redirect"
      - "traefik.http.routers.nginx-mailcow-secure.entrypoints=https"
      - "traefik.http.routers.nginx-mailcow-secure.rule=Host(`mail.example.com`)" 
      - "traefik.http.routers.nginx-mailcow-secure.tls=true"
      - "traefik.http.routers.nginx-mailcow-secure.certresolver=http"
      - "traefik.http.routers.nginx-mailcow-secure.service=nginx-mailcow"
      - "traefik.http.services.nginx-mailcow.loadbalancer.server.port=8080"
      - "traefik.docker.network=proxy"
    networks:
      proxy:

  certdumper:
    image: humenius/traefik-certs-dumper
    container_name: traefik_certdumper
    network_mode: none
    command: --restart-containers mailcowdockerized_postfix-mailcow_1,mailcowdockerized_dovecot-mailcow_1
    volumes:
      - <path_to_acme.json_file_dir>:/traefik:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./data/assets/ssl:/output:rw
    environment:
      - DOMAIN=example.com 

For people who are setting up for the first time, I had to make some additional changes beforehand.

1. Firstly, when you run generate.sh file then in mailcow.conf file you need to make following changes:

   HTTP_PORT=8080
   HTTP_BIND=127.0.0.1

   HTTPS_PORT=8443
   HTTPS_BIND=127.0.0.1

   SKIP_LETS_ENCRYPT=y
   SKIP_CLAMD=y

   We make these changes as we can not run mailcow nginx on the same ports as traefik.

2. Now as nginx-mailcow will be running on 8080 or 8443 so we need to expose one of these ports so traefik can talk to mailcow-nginx service. I already exposed port 8080 in the override compose file)

3. You also need to also adapt your loadbalancer port from 80 to 8080. (As I configured above)
4. You need to also tell which certificate resolver should it use. So you need to add this line in labels (I made this as well above in override config)
5. You have to make sure that your acme.json file (certificate file is accessible by certdumper service). So replace to actual path of acme.json directory path

I hope this helps.