TLS 不适用于 Kubernetes 中的 LoadBalancer 支持的服务

Question

我正在尝试通过将服务类型创建为负载均衡器来公开集群中的应用程序。 这样做的原因是我希望这个应用程序有一个单独的通信渠道。 我有一个 KOPS 集群。 我想使用 AWS 的网络负载均衡器，以便获得静态 IP。 当我创建端口 80 映射到应用程序运行的端口的服务时，一切正常，但是当我尝试添加端口 443 时，它只是超时。

这是有效的配置 -

apiVersion: v1
metadata:
  name: abc
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  labels:
    app: abc
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 9050
  selector:
    app: abc
  type: LoadBalancer

一旦我在配置文件中添加 TLS 支持并部署它。 与负载平衡器的连接超时。 如何向负载均衡器添加 TLS 支持？ 我想通过服务而不是通过入口来做到这一点。 这是对我不起作用的配置，当我在浏览器中粘贴链接时，它超时了。

kind: Service
apiVersion: v1
metadata:
  name: abc
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: xxxxx
  labels:
    app: abc
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 443
    protocol: TCP
    targetPort: 9050
  selector:
    app: abc
  type: LoadBalancer

Answer 1

您可以使用 tls & ssl 终止

   apiVersion: v1
    kind: Service
    metadata:
      name: test-service
      annotations:
        # Note that the backend talks over HTTP.
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
        # TODO: Fill in with the ARN of your certificate.
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:{region}:{user id}:certificate/{id}
        # Only run SSL on the port named "https" below.
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    spec:
      selector:
        app: test-pod
      ports:
      - name: http
        port: 80
        targetPort: 8080
      - name: https
        port: 443
        targetPort: 8080
      type: LoadBalancer

您可以在 aws 证书管理器中添加 tls certficate 并使用证书的 arn 地址到 kubernetes 服务。

就像在 becked 中，您可以终止 https 连接并仅使用 HTTP。

你也可以看看这个： https : //aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/

https://github.com/kubernetes/kubernetes/issues/73297

编辑：1

service.beta.kubernetes.io/aws-load-balancer-type: nlb

如果不起作用，请尝试根据您的负载均衡器类型添加此注释。

Answer 2

您现在可以使用 NLB 和 SSL 终止（NLB 中的 https > 服务中的 http）部署入口。 终于找到了对我有用的解决方案，您可以尝试部署以下 ingress.yaml（确保在部署部分更新您的证书 ARN）：

    ---
# Source: nginx-ingress/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
---
# Source: nginx-ingress/templates/default-backend-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-backend
---
# Source: nginx-ingress/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    component: "controller"
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-controller
data:
  server-snippet: |
    listen 8000;
    if ( $server_port = 80 ) {
       return 308 https://$host$request_uri;
    }
  ssl-redirect: "false"
---
# Source: nginx-ingress/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
---
# Source: nginx-ingress/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress
subjects:
  - kind: ServiceAccount
    name: nginx-ingress
    namespace: ingress-nginx
---
# Source: nginx-ingress/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader-nginx
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - create
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: nginx-ingress/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress
subjects:
  - kind: ServiceAccount
    name: nginx-ingress
    namespace: ingress-nginx
---
# Source: nginx-ingress/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:##REPLACE WITH YOUR CERT ARN"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    component: "controller"
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-controller
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: special
  selector:
    app: nginx-ingress
    release: nginx-ingress
    app.kubernetes.io/component: controller
  type: "LoadBalancer"
---
# Source: nginx-ingress/templates/default-backend-service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    component: "default-backend"
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-default-backend
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
  selector:
    app: nginx-ingress
    release: nginx-ingress
    app.kubernetes.io/component: default-backend
  type: "ClusterIP"
---
# Source: nginx-ingress/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
    app.kubernetes.io/component: controller
  name: nginx-ingress-controller
  annotations:
    {}
spec:
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  replicas: 1
  revisionHistoryLimit: 10
  strategy:
    {}
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app: nginx-ingress
        release: nginx-ingress
        component: "controller"
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      containers:
        - name: nginx-ingress-controller
          image: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/nginx-ingress-default-backend
            - --publish-service=$(POD_NAMESPACE)/nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/nginx-ingress-controller
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: special
              containerPort: 8000
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          resources:
            {}
      hostNetwork: false
      serviceAccountName: nginx-ingress
      terminationGracePeriodSeconds: 60
---
# Source: nginx-ingress/templates/default-backend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
    app.kubernetes.io/component: default-backend
  name: nginx-ingress-default-backend
spec:
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  replicas: 1
  revisionHistoryLimit: 10
  template:
    metadata:
      labels:
        app: nginx-ingress
        release: nginx-ingress
        app.kubernetes.io/component: default-backend
    spec:
      containers:
        - name: nginx-ingress-default-backend
          image: "k8s.gcr.io/defaultbackend-amd64:1.5"
          imagePullPolicy: "IfNotPresent"
          args:
          securityContext:
            runAsUser: 65534
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            {}
      serviceAccountName: nginx-ingress-backend
      terminationGracePeriodSeconds: 60

Answer 3

您的注释是指https端口

service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

但是您的端口名为http ，请更改为https

spec:
  externalTrafficPolicy: Local
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 9050