Not able to access kubernetes api from inside a pod container

Question

I have created a hashicorp vault deployment and configured kubernetes auth. The vault container calls kubernetes api internally from the pod to do k8s authentication, and that call is failing with 500 error code (connection refused). I am using docker for windows kubernetes.

I added the below config to vault for kubernetes auth mechanism.

payload.json

{
    "kubernetes_host": "http://kubernetes",
    "kubernetes_ca_cert": <k8s service account token>
}

curl --header "X-Vault-Token: <vault root token>" --request POST --data @payload.json http://127.0.0.1:8200/v1/auth/kubernetes/config

I got 204 response as expected.

And I created a role for kubernetes auth using which I am trying to login to vault:

payload2.json

{
    "role": "tanmoy-role",
    "jwt": "<k8s service account token>"
}

curl --request POST --data @payload2.json http://127.0.0.1:8200/v1/auth/kubernetes/login

The above curl is giving below response:

{"errors":["Post http://kubernetes/apis/authentication.k8s.io/v1/tokenreviews : dial tcp 10.96.0.1:80: connect: connection refused"]}

Below is my kubernetes service up and running properly and I can also access kubernetes dashboard by using proxy.

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP              PORT(S)                         AGE
kubernetes      ClusterIP      10.96.0.1       <none>                   443/TCP                         13d

I am not able to figure out why 'kubernetes' service is not accessible from inside the container. Any help would be greatly appreciated.

Edit 1. My vault pod and service are working fine:

service

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP              PORT(S)                         AGE
vault-elb-int   LoadBalancer   10.104.197.76   localhost,192.168.0.10   8200:31650/TCP,8201:31206/TCP   26h

Pod

NAME                     READY   STATUS    RESTARTS   AGE
vault-84c65db6c9-pj6zw   1/1     Running   0          21h

Edit 2. As John suggested, I changed the 'kubernetes_host' in payload.json to ' https://kubernetes '. But now I am getting this error:

{"errors":["Post https://kubernetes/apis/authentication.k8s.io/v1/tokenreviews: x509: certificate signed by unknown authority"]}

Answer 1

Your login request is being sent to the tokenreview endpoint on port 80. I think this is because your kubernetes_host specifies a http URL. The 500 response is because it's not listening on port 80, but on 443 instead (as you can see in your service list output).

Try changing to https when configuring the auth, ie

payload.json

{
    "kubernetes_host": "https://kubernetes",
    "kubernetes_ca_cert": <k8s service account token>
}

Answer 2

Finally I have figured out what went wrong:

my payload.json content was wrong

it should be like this:

{
      "kubernetes_host": "https://kubernetes",
      "kubernetes_ca_cert": <kubectl exec to vault pod and cat  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, now make the cert one line by following this answer: https://stackoverflow.com/a/14580203/2054147>
}

Now below endpoint is working fine and returning the desire client_token

curl --request POST --data @payload2.json http://127.0.0.1:8200/v1/auth/kubernetes/login

Thanks @John for helping me to figure out the initial issue with kubernetes_host.