stackoom
  简体   繁体   中英

Unable to load kubernetes dashboard after successful oauth2

 原文  2022-01-04 19:37:14  4   1    kubernetes/ kubernetes-dashboard/ oauth2-proxy

Question

I have deployment my kubernetes cluster v1.23.1 with kubeadm and configured it with the keycloak identity provider for authentication.

API server configuration for keycloak IDP

...
    - --oidc-issuer-url=https://kubemaster:8443/auth/realms/local
    - --oidc-client-id=gatekeeper
    - --oidc-username-claim=name
    - --oidc-groups-claim=groups
    - --oidc-ca-file=/etc/kubernetes/ssl/kubemaster.crt
...

I am running oauth2-proxy in another pod which authenticates with keycloak idp and provides the tokens (id_token). Here is my oauth2-proxy deployment with the service to expose it.

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
        - name: oauth2-proxy
          image: quay.io/oauth2-proxy/oauth2-proxy:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 4180
              protocol: TCP
          args:
          - --provider=oidc
          - --email-domain=*
          - --http-address=0.0.0.0:4180
          env:
          - name: OAUTH2_PROXY_OIDC_ISSUER_URL
            value: https://192.168.122.54:8443/auth/realms/local
          - name: OAUTH2_PROXY_REDIRECT_URL
            value: https://kubernetes-dashboard.localdev.me:8081/oauth2/callback
          - name: OAUTH2_PROXY_CLIENT_ID
            value: gatekeeper
          - name: OAUTH2_PROXY_CLIENT_SECRET
            value: jZzvJ0wCDDwltV3tAf0SXSbVoKXM1RqV
          - name: OAUTH2_PROXY_COOKIE_SECRET
            value: kgKUT3IMmESA81VWXvRpYIYwMSo1xndwIogUks6IS00=
          - name: OAUTH2_PROXY_UPSTREAM
            value: https://kubernetes-dashboard
          - name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
            value: "true"
          - name: OAUTH2_PROXY_COOKIE_DOMAIN
            value: .localdev.me
          - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
            value: "true"
          - name: OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER
            value: "true"
          - name: OAUTH2_PROXY_SSL_UPSTREAM_INSECURE_SKIP_VERIFY
            value: "true"
          - name: OAUTH2_PROXY_OIDC_EMAIL_CLAIM
            value: email
          - name: OAUTH2_PROXY_GROUPS_CLAIM
            value: groups
          - name: OAUTH2_PROXY_ALLOWED_GROUPS
            value: developers

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  #type: NodePort
  ports:
    - port: 4180
      targetPort: 4180
      name: http
  selector:
    k8s-app: oauth2-proxy

I have nginx ingress deployed in front to route the request to oauth2-proxy as shown below.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: oauth-proxy
    namespace: kubernetes-dashboard
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: kubernetes-dashboard.localdev.me
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: oauth2-proxy
            port:
              number: 4180
  tls:
  - hosts:
    - kubernetes-dashboard.localdev.me
    secretName: kubernetes-dashboard-ingress-tls

I am using port forward from localhost to ingress resource on port 443.

kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8081:443

The issue currently i am facing is i am able to launch the url - https://kubernetes-dashboard.localdev.me:8081/ which routes to oauth2 proxy page from there i am able to launch the keycloak realm login page which i can authenticate. But once the keycloak authentication is successfully done i am not able to load the upstream which is kubernetes-dashboard service with the id_token. Instead i am getting a 404 not found for https://kubernetes-dashboard.localdev.me:8081/

oauth2-proxy Log details 

[2022/01/04 19:06:24] [oauthproxy.go:866] No valid authentication in request. Initiating login.
192.168.1.169:33826 - 48490760d6e0fc519e13d3158878112a - - [2022/01/04 19:06:24] kubernetes-dashboard.localdev.me:8081 GET - "/" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 403 8033 0.000
[2022/01/04 19:06:25] [oauthproxy.go:866] No valid authentication in request. Initiating login.
192.168.1.169:33826 - a87fbdc228026d99e6fc9f29d9ffc6df - - [2022/01/04 19:06:25] kubernetes-dashboard.localdev.me:8081 GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 403 8044 0.000
192.168.1.169:33826 - 3c052e49f9f77e89c561f3d0bef47fed - - [2022/01/04 19:06:43] kubernetes-dashboard.localdev.me:8081 GET - "/oauth2/start?rd=%2F" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 302 356 0.000
192.168.1.169:33826 - a16b9b1b22ec03e58b9b930caa325f5b - alice@stack.com [2022/01/04 19:06:55] [AuthSuccess] Authenticated via OAuth2: Session{email:alice@stack.com user:52ed0190-4d9c-4749-90f4-37e5896bdc42 PreferredUsername:alice token:true id_token:true created:2022-01-04 19:06:55.77587057 +0000 UTC m=+290.895007232 expires:2022-01-04 19:11:55.775499536 +0000 UTC m=+590.894636200 refresh_token:true groups:[developers]}
192.168.1.169:33826 - a16b9b1b22ec03e58b9b930caa325f5b - - [2022/01/04 19:06:55] kubernetes-dashboard.localdev.me:8081 GET - "/oauth2/callback?state=HBDoNuX3mudlwQrjdEkxYP9yjgHn5_mqOXk8T-G21dg%3A%2F&session_state=e928739a-c94d-49ba-80a9-a1f83a936bf1&code=9a592559-236c-4896-86b4-ba8f657821e6.e928739a-c94d-49ba-80a9-a1f83a936bf1.b7b62a18-269b-480a-b5bd-ad3c16f94394" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 302 24 0.030
192.168.1.169:33826 - 9288833c2f8e82f2ad94c6260bcf68c7 - alice@stack.com [2022/01/04 19:06:55] kubernetes-dashboard.localdev.me:8081 GET - "/" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 404 19 0.000

Ingress controller logs

127.0.0.1 - - [04/Jan/2022:19:02:26 +0000] "GET / HTTP/2.0" 404 19 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 2842 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 19 0.000 404 d19c25394498799f5024567ddd1fafcd
127.0.0.1 - - [04/Jan/2022:19:06:24 +0000] "GET / HTTP/2.0" 403 8033 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 297 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 8060 0.000 403 48490760d6e0fc519e13d3158878112a
127.0.0.1 - - [04/Jan/2022:19:06:25 +0000] "GET /favicon.ico HTTP/2.0" 403 8044 "https://kubernetes-dashboard.localdev.me:8081/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 102 0.003 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 8071 0.004 403 a87fbdc228026d99e6fc9f29d9ffc6df
127.0.0.1 - - [04/Jan/2022:19:06:43 +0000] "GET /oauth2/start?rd=%2F HTTP/2.0" 302 356 "https://kubernetes-dashboard.localdev.me:8081/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 32 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 356 0.000 302 3c052e49f9f77e89c561f3d0bef47fed
127.0.0.1 - - [04/Jan/2022:19:06:55 +0000] "GET /oauth2/callback?state=HBDoNuX3mudlwQrjdEkxYP9yjgHn5_mqOXk8T-G21dg%3A%2F&session_state=e928739a-c94d-49ba-80a9-a1f83a936bf1&code=9a592559-236c-4896-86b4-ba8f657821e6.e928739a-c94d-49ba-80a9-a1f83a936bf1.b7b62a18-269b-480a-b5bd-ad3c16f94394 HTTP/2.0" 302 24 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 351 0.031 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 24 0.028 302 a16b9b1b22ec03e58b9b930caa325f5b
127.0.0.1 - - [04/Jan/2022:19:06:55 +0000] "GET / HTTP/2.0" 404 19 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 2827 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 19 0.004 404 9288833c2f8e82f2ad94c6260bcf68c7

Please suggest on how i can route the authenticated session with id_token to kubernetes dashboard service for loading the dashboard.

1 answers

solution1
 0  2022-01-14 04:11:43

I was finally able to resolve my issue with some updates to my yaml definition files.

Assuming you have a kubernetes cluster installed v1.23.1 with kubeadm on Ubuntu 20.04 and setup networking with flannel networking --pod-network-cidr=10.244.0.0/16. Also you have the keycloak oidc service setup (image - quay.io/keycloak/keycloak:16.1.0). Here are the updated yml definition files which helped me to resolve this issue.

Ingress controller applied -

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/cloud/deploy.yaml

kubernetes dashboard applied - 

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yaml

oauth2-proxy.yml 

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
      - args:
        - --provider=oidc
        - --email-domain=*
        - --http-address=0.0.0.0:4180
        env:
        - name: OAUTH2_PROXY_OIDC_ISSUER_URL
          value: https://kubemaster:8443/auth/realms/local
        - name: OAUTH2_PROXY_REDIRECT_URL
          value: https://<FQDN>/oauth2/callback
        - name: OAUTH2_PROXY_CLIENT_ID
          value: gatekeeper
        - name: OAUTH2_PROXY_CLIENT_SECRET
          value: jZzvJ0wCDDwltV3tAf0SXSbVoKXM1RqV
        - name: OAUTH2_PROXY_COOKIE_SECRET
          value: kgKUT3IMmESA81VWXvRpYIYwMSo1xndwIogUks6IS00=
        - name: OAUTH2_PROXY_UPSTREAM
          value: https://kubernetes-dashboard
        - name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
          value: "true"
        - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
          value: "true"
        - name: OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER
          value: "true"
        - name: OAUTH2_PROXY_SSL_UPSTREAM_INSECURE_SKIP_VERIFY
          value: "true"
        - name: OAUTH2_PROXY_OIDC_EMAIL_CLAIM
          value: email
        - name: OAUTH2_PROXY_GROUPS_CLAIM
          value: groups
        - name: OAUTH2_PROXY_ALLOWED_GROUPS
          value: developers
        - name: OAUTH2_PROXY_SKIP_PROVIDER_BUTTON
          value: "true"
        - name: OAUTH2_PROXY_SET_AUTHORIZATION_HEADER
          value: "true"
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        imagePullPolicy: Always
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP

dashboard-ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/auth-response-headers: "authorization"
  name: external-auth-oauth2
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
  - host: __INGRESS_HOST__
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443
---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
  - host: __INGRESS_HOST__
    http:
      paths:
      - path: /oauth2
        pathType: Prefix
        backend:
          service:
            name: oauth2-proxy
            port:
              number: 4180
  tls:
  - hosts:
    - __INGRESS_HOST__
    secretName: __INGRESS_SECRET__

The issue with keycloak oidc provider its unable to set the Authroization header for which we need to manaully add the set authorization header in the oauth2-proxy defintion and also this header need to be appended in the response from the ingress to the upstream kubernetes dashboard by setting annotation - nginx.ingress.kubernetes.io/auth-response-headers: "authorization"

Also adding a reference for kubernetes dashboard authentication using github oauth2 which i was following to resolve this issue.

https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/#example-oauth2-proxy-kubernetes-dashboard

暂无
暂无

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

Related Question Unable to access Kubernetes Dashboard Unable to get kubernetes dashboard Kubernetes: Unable to access to kubernetes dashboard Kubernetes Google OAuth2 Sign In kubernetes dashboard will not load Unable to install Kubernetes dashboard on AWS Unable to open the kubernetes dashboard in Azure Kubernetes Service Kubernetes Dashboard not working after configuring it Unable to access the Kubernetes Dashboard on Google Cloud Platform Unable to expose kubernetes dashboard to access it from outside
Related Tags
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM