Permission denied when authenticating pod to external vault service running on gke

Question

GKE version - 1.14 Currently I have two private gke cluster ( Vault cluster and app cluster)

Getting following errors:

vault errors -
auth.kubernetes.auth_kubernetes_b0f01fa6: login unauthorized due to: Post "https://10.V.V.194:443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp `10.V.V.194`:443: i/o timeout

-> where

10.V.V.194 -- is master IP address (no https://) via `kubectl cluster-info

Application pod logs

    * permission denied" backoff=1.324573453
2020-10-12T14:39:46.421Z [INFO]  auth.handler: authenticating
2020-10-12T14:40:16.427Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://10.LB.LB.38:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied" backoff=2.798763368

-> Where

http://10.LB.LB.38:8200 is Internal LB IP 

Vault setup

    NAME   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         
test-vault  LoadBalancer   240.130.0.59   10.LB.LB.38   8200:32105/TCP,8201:31147/TCP  

How is K8s auth methods enable

    $ export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo) 

# determine Kubernetes master IP address (no https://) via `kubectl cluster-info`
$ export K8S_HOST=<K8S_MASTER_IP>   ----- App cluster ip 

# set VAULT_TOKEN & VAULT_ADDR before next steps
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="https://$K8S_HOST:443" \
        kubernetes_ca_cert="$SA_CA_CRT"

How is vault inject setup in application cluster

name: AGENT_INJECT_VAULT_ADDR
      value: http://10.LB.LB.38:8200

Cluster B ( app cluster )

kubectl create serviceaccount vault-auth -n default
-----
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default

vault auth enable kubernetes
-----------
vault write auth/kubernetes/config kubernetes_host="${K8S_HOST}"   
    kubernetes_ca_cert="${VAULT_SA_CA_CRT}" 
    token_reviewer_jwt="${TR_ACCOUNT_TOKEN}"
-----------
vault secrets enable -path=secret/ kv
-----------
vault policy write myapp-kv-rw - <<EOF
 path "secret/myapp/*" {
 capabilities = ["create", "read", "update", "delete", "list"]
}
--------------
vault write auth/kubernetes/role/myapp-role \
 bound_service_account_names=default \
 bound_service_account_namespaces=default \
 policies=default,myapp-kv-rw \
 ttl=15m

Can you Please let me know, If I miss anything ?

Answer 1

You can try to manually access the Kubernetes API (in your app cluster) from your Vault cluster with the same configuration you used to setup Vault.

curl -X "POST" "${K8S_HOST}/apis/authentication.k8s.io/v1/tokenreviews" \
     --cacert <(echo $VAULT_SA_CA_CRT)
     -H 'Authorization: Bearer ${TR_ACCOUNT_TOKEN}' \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{
  "kind": "TokenReview",
  "apiVersion": "authentication.k8s.io/v1",
  "spec": {
    "token": "${INTERNAL_APP_TOKEN}"
  }
}'

If that doesn't work, Vault won't be able to call the API to get the tokens reviewed. If you get a connection timed out or refused, it's most likely a firewall issue. If you get unauthorized, your cluster role setup is probably wrong.

Answer 2

Curious as to which version of k8s you're using. I was having the same issue using v1.21.1 . I had to add an issuer to the per the docs ( https://www.vaultproject.io/docs/auth/kubernetes )

Kubernetes 1.21+ clusters may require setting the service account issuer to the same value as kube-apiserver's --service-account-issuer flag. This is because the service account JWTs for these clusters may have an issuer specific to the cluster itself, instead of the old default of kubernetes/serviceaccount

like so

vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  issuer="\"test-aks-cluster-dns-d6cbb78e.hcp.uksouth.azmk8s.io\""

and the issuer can be obtained by running kubectl proxy & curl --silent http://127.0.0.1:8001/.well-known/openid-configuration | jq -r .issuer kubectl proxy & curl --silent http://127.0.0.1:8001/.well-known/openid-configuration | jq -r .issuer