使用 Prometheus 适配器清空自定义指标资源列表

Question

I have a minikube installation on my windows machine. My application exposes a custom metric 'http_requests_total'. I installed Prometheus operator first and configured it scrape the custom metrics. I can see the custom metric appear in Prometheus dashboard.

Trouble arises when I install Prometheus Adapter. I use following helm command to install the adapter:

helm install my-release prometheus-community/prometheus-adapter

Then I run following command to edit the configmap and add rule for my custom metric:

kubectl edit cm my-release-prometheus-adapter

I add following section in this configmap:

- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

Having done this, when I execute the following command:

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

I do not see my custom resource being read:

{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[]}

I have read multiple similar questions posted on stackoverflow but none seems to have the working answer. I tried to change the prometheus url for my adapter but didn't succeed with that. What am I missing? Following is the log from the adapter pod:

I0706 22:41:15.240788       1 adapter.go:101] successfully using in-cluster auth
E0706 22:41:15.274396       1 provider.go:227] unable to update list of all metrics: unable to fetch metrics for query "{namespace!=\"\",__name__!~\"^container_.*\"}": Get "http://prometheus-kube-prometheus-prometheus.prom.svc.cluster.local:9090/api/v1/series?match%5B%5D=%7Bnamespace%21%3D%22%22%2C__name__%21~%22%5Econtainer_.%2A%22%7D&start=1625611215.271": dial tcp: lookup prometheus-kube-prometheus-prometheus.prom.svc.cluster.local on 10.96.0.10:53: no such host
I0706 22:41:15.636300       1 serving.go:325] Generated self-signed cert (/tmp/cert/apiserver.crt, /tmp/cert/apiserver.key)
I0706 22:41:15.637101       1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key"
I0706 22:41:16.192597       1 requestheader_controller.go:244] Loaded a new request header values for RequestHeaderAuthRequestController
I0706 22:41:16.193884       1 config.go:655] Not requested to run hook priority-and-fairness-config-consumer
I0706 22:41:16.249643       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0706 22:41:16.249689       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0706 22:41:16.249807       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0706 22:41:16.249829       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0706 22:41:16.249928       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0706 22:41:16.249977       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0706 22:41:16.250288       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key
I0706 22:41:16.250322       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:172
I0706 22:41:16.250390       1 reflector.go:255] Listing and watching *v1.ConfigMap from k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:172
I0706 22:41:16.250322       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206
I0706 22:41:16.250536       1 reflector.go:255] Listing and watching *v1.ConfigMap from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206
I0706 22:41:16.250555       1 reflector.go:219] Starting reflector *v1.ConfigMap (12h0m0s) from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206
I0706 22:41:16.250608       1 reflector.go:255] Listing and watching *v1.ConfigMap from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206
I0706 22:41:16.250577       1 tlsconfig.go:200] loaded serving cert ["serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key"]: "localhost@1625611275" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer="localhost-ca@1625611275" (2021-07-06 21:41:15 +0000 UTC to 2022-07-06 21:41:15 +0000 UTC (now=2021-07-06 22:41:16.2505337 +0000 UTC))
I0706 22:41:16.250950       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1625611276" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1625611275" (2021-07-06 21:41:15 +0000 UTC to 2022-07-06 21:41:15 +0000 UTC (now=2021-07-06 22:41:16.2509375 +0000 UTC))
I0706 22:41:16.251003       1 secure_serving.go:197] Serving securely on [::]:6443
I0706 22:41:16.251321       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0706 22:41:16.349970       1 shared_informer.go:270] caches populated
I0706 22:41:16.350023       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I0706 22:41:16.350129       1 shared_informer.go:270] caches populated
I0706 22:41:16.350142       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0706 22:41:16.350164       1 shared_informer.go:270] caches populated
I0706 22:41:16.350186       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0706 22:41:16.350698       1 tlsconfig.go:178] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "minikubeCA" [client,serving] issuer="<self>" (2021-06-07 20:39:52 +0000 UTC to 2031-06-06 20:39:52 +0000 UTC (now=2021-07-06 22:41:16.3506808 +0000 UTC))
I0706 22:41:16.350796       1 tlsconfig.go:178] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "front-proxy-ca" [] issuer="<self>" (2021-06-08 20:39:52 +0000 UTC to 2031-06-06 20:39:52 +0000 UTC (now=2021-07-06 22:41:16.3507595 +0000 UTC))
I0706 22:41:16.351138       1 tlsconfig.go:200] loaded serving cert ["serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key"]: "localhost@1625611275" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer="localhost-ca@1625611275" (2021-07-06 21:41:15 +0000 UTC to 2022-07-06 21:41:15 +0000 UTC (now=2021-07-06 22:41:16.3511272 +0000 UTC))
I0706 22:41:16.351349       1 named_certificates.go:53] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1625611276" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1625611275" (2021-07-06 21:41:15 +0000 UTC to 2022-07-06 21:41:15 +0000 UTC (now=2021-07-06 22:41:16.3513364 +0000 UTC))
I0706 22:41:26.255679       1 reflector.go:530] k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:172: Watch close - *v1.ConfigMap total 0 items received
I0706 22:41:26.256047       1 reflector.go:530] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Watch close - *v1.ConfigMap total 0 items received
I0706 22:41:26.257735       1 reflector.go:530] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Watch close - *v1.ConfigMap total 0 items received

Following are the arguments in the deployment for the adapter:

Args:
      /adapter
      --secure-port=6443
      --cert-dir=/tmp/cert
      --logtostderr=true
      --prometheus-url=http://prometheus.default.svc:9090
      --metrics-relist-interval=1m
      --v=6
      --config=/etc/adapter/config.yaml

Following are various services present on my local cluster:

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   2d6h
kubernetes                                ClusterIP   10.96.0.1        <none>        443/TCP                      28d
my-release-prometheus-adapter             ClusterIP   10.106.147.10    <none>        443/TCP                      23m
prometheus-grafana                        ClusterIP   10.108.144.246   <none>        80/TCP                       2d6h
prometheus-kube-prometheus-alertmanager   ClusterIP   10.108.19.130    <none>        9093/TCP                     2d6h
prometheus-kube-prometheus-operator       ClusterIP   10.108.170.84    <none>        443/TCP                      2d6h
prometheus-kube-prometheus-prometheus     ClusterIP   10.98.67.168     <none>        9090/TCP                     2d6h
prometheus-kube-state-metrics             ClusterIP   10.97.252.48     <none>        8080/TCP                     2d6h
prometheus-operated                       ClusterIP   None             <none>        9090/TCP                     2d6h
prometheus-prometheus-node-exporter       ClusterIP   10.107.194.240   <none>        9100/TCP                     2d6h
sample-app                                ClusterIP   10.97.86.241     <none>        80/TCP                       4h10m

In the beginning of logs for the adapter pod, I see following line of error:

unable to update list of all metrics: unable to fetch metrics for query "{__name__=~\"^container_.*\",container!=\"POD\",namespace!=\"\",pod!=\"\"}": Get "http://prometheus.default.svc:9090/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Econtainer_.%2A%22%2Ccontainer%21%3D%22POD%22%2Cnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1625614091.754": dial tcp: lookup prometheus.default.svc on 10.96.0.10:53: no such host

Answer 1

The prometheus-adapter is not able to connect to your prometheus instance, this line in the log indicates as much:

E0706 22:41:15.274396       1 provider.go:227] unable to update list of all metrics: unable to fetch metrics for query "{namespace!=\"\",__name__!~\"^container_.*\"}": Get "http://prometheus-kube-prometheus-prometheus.prom.svc.cluster.local:9090/api/v1/series?match%5B%5D=%7Bnamespace%21%3D%22%22%2C__name__%21~%22%5Econtainer_.%2A%22%7D&start=1625611215.271": dial tcp: lookup prometheus-kube-prometheus-prometheus.prom.svc.cluster.local on 10.96.0.10:53: no such host

Although you have a service called prometheus-kube-prometheus-prometheus , the url is expecting that service to live in the prom namespace. You should edit the prometheus url (in the same way you added the series query) to have the proper url. If this service is in the default namespace you can specify prometheus-kube-prometheus-prometheus.svc.cluster.local otherwise it would be prometheus-kube-prometheus-prometheus.<NAMESPACE>.svc.cluster.local