“ERR_TOO_MANY_REDIRECTS”nginx-ingress controller 不會覆蓋 X-Forwarded-Proto：http，X-Forwarded-Scheme：http

Question

我們正在使用 IAP，GCP L7 負載均衡器和 nginx-ingress controller（版本 0.49.3）。 我們部署了自托管 GitLab，我們收到“ERR_TOO_MANY_REDIRECTS”。

經過幾天緊張的故障排除后，我們注意到

POST /api/v4/jobs/request HTTP/1.1
Host: gitlab.ci.g.nakhoda.ai
X-Request-ID: dfa874d97ed06e0e7a7cf17c0a4ae2c0
X-Real-IP: 35.191.12.183
X-Forwarded-For: 35.191.12.183
X-Forwarded-Host: gitlab.ci.g.nakhoda.ai
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Scheme: http
X-Original-Forwarded-For: 62.189.73.245, 35.186.225.221
Content-Length: 714
User-Agent: gitlab-runner 15.6.1 (15-6-stable; go1.18.8; linux/amd64)
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip
X-Cloud-Trace-Context: 4efdd65224a5882999fd0fb26a888bfd/2273898499790060164
Via: 1.1 google

X-Forwarded-Proto: http 和 X-Forwarded-Scheme: http 。 在擺弄了一下之后，我們在 nginx-ingress-controller 的容器中執行並編輯nginx.conf為它們都設置了 https 並添加了注釋ssl-redirect: false 。

之后請求看起來像這樣：

POST /api/v4/jobs/request HTTP/1.1
Host: gitlab.ci.g.nakhoda.ai
X-Request-ID: f103bba96d47527dae15087d1dd1d476
X-Real-IP: 35.191.12.180
X-Forwarded-For: 35.191.12.180
X-Forwarded-Host: gitlab.ci.g.nakhoda.ai
X-Forwarded-Port: 80
X-Forwarded-Proto: https
X-Forwarded-Scheme: https
X-Scheme: http
X-Original-Forwarded-For: 194.203.216.4, 35.186.225.221
Content-Length: 714
User-Agent: gitlab-runner 15.6.1 (15-6-stable; go1.18.8; linux/amd64)
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip
X-Cloud-Trace-Context: 3ceb1d9d95fa28be4da929dea1f3ac95/3016269448751760533
Via: 1.1 google

通過手動編輯，我們現在可以毫無問題地訪問 GitLab，但問題是這是我們添加的手動修復，因此每次管道進行部署時，都會將其重置到位。

問題是我們希望添加自定義標頭以覆蓋 [nginx.conf]( https://kube.netes.github.io/ingress-nginx/examples/customization/custom-headers/不會覆蓋正常的 nginx。 conf 配置），但是在添加具有特定內容的 ConfigMap 之后，或者只是將類似的注釋添加到 nginx-ingress controller 的 Ingress 或 ConfigMap 中：

Gitlab 入口

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab-chart-webservice-default
  namespace: gitlab
  labels:
    app: webservice
    app.kubernetes.io/managed-by: Helm
    chart: webservice-6.6.2
    gitlab.com/webservice-name: default
    heritage: Helm
    release: gitlab-chart
  annotations:
    kubernetes.io/ingress.provider: nginx
    meta.helm.sh/release-name: gitlab-chart
    meta.helm.sh/release-namespace: gitlab
    nginx.ingress.kubernetes.io/proxy-body-size: 512m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '15'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'
    nginx.ingress.kubernetes.io/service-upstream: 'true'
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
status:
  loadBalancer:
    ingress:
      - ip: 1.1.1.1
      - ip: 1.1.1.1
spec:
  ingressClassName: stable-protected
  tls:
    - hosts:
        - gitlab.ci.example.com
      secretName: gitlab-cert
  rules:
    - host: gitlab.ci.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gitlab-chart-webservice-default
                port:
                  number: 8181

ConfigMap nginx controller

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
  namespace: ingress-stable-protected
  annotations:
    refresh-me: |
      This 'refresh-me' annotation is purely used as a placeholder so we can
      modify its value in order to manually force the nginx ingress-controller
      to reload its configuration.
      Reload.
data:
  enable-vts-status: 'true'
  hide-headers: Strict-Transport-Security
  hsts: 'true'
  hsts-include-subdomains: 'false'
  hsts-preload: 'false'
  proxy-body-size: 2048m
  proxy-buffer-size: 16k
  proxy-connect-timeout: '5'
  proxy-next-upstream: 'off'
  proxy-read-timeout: '600'
  proxy-send-timeout: '600'
  server-name-hash-bucket-size: '256'
  server-tokens: 'false'

nginx.ingress.kubernetes.io/x-forwarded-proto=https
nginx.ingress.kubernetes.io/x-forwarded-scheme=https

它仍然需要 nginx.conf http所以我們有點堅持這個臨時修復。

也嘗試添加一堆自定義標頭和 x-forwarded-headers 但無濟於事。

Answer 1

問題是在 nginx-ingress 的 0.24.0 版本之后，注釋發生了變化，因此通過將以下內容添加到 nginx configmap 中，不再有錯誤。

可以在這張票中找到更深入的描述

use-forwarded-headers: "true"
use-proxy-protcol: "false"

完整示例：

apiVersion: v1
data:
  enable-vts-status: "true"
  hide-headers: Strict-Transport-Security
  hsts-preload: "false"
  proxy-body-size: 2048m
  proxy-buffer-size: 16k
  proxy-connect-timeout: "5"
  proxy-next-upstream: "off"
  proxy-read-timeout: "600"
  proxy-send-timeout: "600"
  server-name-hash-bucket-size: "256"
  server-tokens: "false"
  use-forwarded-headers: "true"
  use-proxy-protcol: "false"
kind: ConfigMap
metadata:
  annotations:
    refresh-me: |
      This 'refresh-me' annotation is purely used as a placeholder so we can
      modify its value in order to manually force the nginx ingress-controller
      to reload its configuration.
  labels:
    app.kubernetes.io/managed-by: pulumi
  name: nginx
  namespace: namespace