How to access Prometheus and Grafana installed in the ingress-nginx namespace on managed k8s running in Azure?

Question

I have followed the instructions at https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/ to install the Prometheus and Grafana to monitor the performance of the ingress-nginx controller.

However, the instructions to access the Prometheus dashboard did not work for me.

Indeed, according to the instructions we should be able to access the service through one of the IPs returned by the following command:

kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\(@.type==\"InternalIP\"\)].address}

However, I do not understand how it is possible, since the IPs are internal to the cluster. Even in their documentation:

How can it possibly work if the IP addresses 10.192.0.2 10.192.0.3 10.192.0.4 are internal to the cluster?

---

Here is what I have:

mark@D-LP0003:~$ kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
grafana                              NodePort       10.0.6.223     <none>          3000:32275/TCP               23m
ingress-nginx-controller             LoadBalancer   10.0.254.117   40.76.145.210   80:32542/TCP,443:31960/TCP   51m
ingress-nginx-controller-admission   ClusterIP      10.0.110.16    <none>          443/TCP                      51m
prometheus-server                    NodePort       10.0.208.16    <none>          9090:32053/TCP               48m
mark@D-LP0003:~$

No external IP. OK, the documentation has it in mind already and suggests to run another command:

mark@D-LP0003:~$ kubectl get nodes -o wide
NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-nodepool1-30294001-vmss000000   Ready    agent   14h   v1.17.9   10.240.0.4    <none>        Ubuntu 16.04.7 LTS   4.15.0-1092-azure   docker://3.0.10+azure
aks-nodepool1-30294001-vmss000001   Ready    agent   14h   v1.17.9   10.240.0.5    <none>        Ubuntu 16.04.7 LTS   4.15.0-1092-azure   docker://3.0.10+azure
aks-nodepool1-30294001-vmss000003   Ready    agent   11h   v1.17.9   10.240.0.6    <none>        Ubuntu 16.04.7 LTS   4.15.0-1092-azure   docker://3.0.10+azure
mark@D-LP0003:~$

Still no external IP. And at this point the documentation suggests to run yet another command:

mark@D-LP0003:~$ kubectl get nodes --selector=kubernetes.io/role!=master -o jsonpath={.items[*].status.addresses[?\(@.type==\"InternalIP\"\)].address}
10.240.0.4 10.240.0.5 10.240.0.6mark@D-LP0003:~$

I get back the same internal IPs.

The only way I found to access the dashboard is through the client side port forwarding:

kubectl port-forward svc/prometheus-server 9090:9090 -n ingress-nginx  &

And then I can access the server at http://localhost:9090 .

But I do not want it. The service is supposed to be externally accessible, so how do I access it?

Answer 1

You have few options

1. Since you are in Azure you could create a LoadBalancer type service instead of NodePort type service to expose grafana and prometheus. You will get an external IP for the service which you can use to access it.

2. Use the Master nodes IP and NodePort to access it. Assuming master nodes have external IP. Use MASTERNODEIP:32053 for prometheus and MASTERNODEIP:32275 for grafana

3. Use ingress resource to expose them. You already have a external IP for the LoadBalancer of the nginx ingress controller. Use that IP to access it. Refer to this to know how to do it.