Prometheus 无法抓取 kong-ingress-controller 指标

Question

I've installed Prometheus and Grafana to monitor my K8S cluster and microservices using helm charts :

helm install monitoring prometheus-community/kube-promehteus-stack --values prometheus-values.yaml --version 16.10.0 --namespace monitoring --create-namespace

the content of promehteus-values.yaml is:

prometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchLabels:
        prometheus: devops
commonLabels:
  prometheus: devops
grafana:
  adminPassword: test123

Then I installed kong-ingress-controller using helm-charts :

helm install kong kong/kong --namespace kong --create-namespace --values kong.yaml --set ingressController.installCRDs=false

the content of kong.yaml file is:

podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8100"

I've also changed the value of metricsBindAdress in kube-proxy configmap to 0.0.0.0:10249 .

then I installed kong prometheus plugin using this yaml file :

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: prometheus
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
plugin: prometheus

My kong endpoint is :

$ kubectl edit endpoints -n kong kong-kong-proxy

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2021-10-27T03:28:25Z"
  creationTimestamp: "2021-10-26T04:44:57Z"
  labels:
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: "2.6"
    enable-metrics: "true"
    helm.sh/chart: kong-2.5.0
  name: kong-kong-proxy
  namespace: kong
  resourceVersion: "6553623"
  uid: 91f2054f-7fb9-4d63-8b65-be098b8f6547
subsets:
- addresses:
  - ip: 10.233.96.41
    nodeName: node2
    targetRef:
      kind: Pod
      name: kong-kong-69fd7d7698-jjkj5
      namespace: kong
      resourceVersion: "6510442"
      uid: 26c6bdca-e9f1-4b32-91ff-0fadb6fce529
  ports:
  - name: kong-proxy
    port: 8000
    protocol: TCP
  - name: kong-proxy-tls
    port: 8443
    protocol: TCP

Finally I wrote the serviceMonitor for kong like this :

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  generation: 1
  labels:
    prometheus: devops
  name: kong-sm
  namespace: kong
spec:
  endpoints:
  - interval: 30s
    port: kong-proxy
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - kong
  selector:
    matchLabels:
      app.kubernetes.io/instance: kong

After all of this ; the targets in prometheus dashboard looks like this:

What did I miss/do wrong?

Answer 1

Let's take a look to the Kong deployment first ( pay extra attention to the bottom of this file ) :

kubectl edit deploy -n kong kong-kong :

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    [...]
  creationTimestamp: "2021-10-26T04:44:58Z"
  generation: 1
  labels:
    app.kubernetes.io/component: app
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: "2.6"
    helm.sh/chart: kong-2.5.0
  name: kong-kong
  namespace: kong
  [...]
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: app
      app.kubernetes.io/instance: kong
      app.kubernetes.io/name: kong
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    [...]
   - env:
        [...]
        image: kong:2.6
        imagePullPolicy: IfNotPresent
        [...]
        name: proxy
        ports:
        - containerPort: 8000
          name: proxy
          protocol: TCP
        - containerPort: 8443
          name: proxy-tls
          protocol: TCP
         ############################################
         ## THIS PART IS IMPORTANT TO US :          #
         ############################################
        - containerPort: 8100 
          name: status
          protocol: TCP
        [...]

As you can see, in the sepc.template.spec.env.ports part we have 3 ports , the 8100 will be used for get metrics so if you can't see this port in the kong endpoint , add it manually to the bottom of kong endpoint:

$ kubectl edit endpoints -n kong kong-kong-proxy :

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2021-10-26T04:44:58Z"
  creationTimestamp: "2021-10-26T04:44:57Z"
  labels:
    app.kubernetes.io/instance: kong
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kong
    app.kubernetes.io/version: "2.6"
    enable-metrics: "true"
    helm.sh/chart: kong-2.5.0
  name: kong-kong-proxy
  namespace: kong
  resourceVersion: "7160332"
  uid: 91f2054f-7fb9-4d63-8b65-be098b8f6547
subsets:
- addresses:
  - ip: 10.233.96.41
    nodeName: node2
    targetRef:
      kind: Pod
      name: kong-kong-69fd7d7698-jjkj5
      namespace: kong
      resourceVersion: "6816178"
      uid: 26c6bdca-e9f1-4b32-91ff-0fadb6fce529
  ports:
  - name: kong-proxy
    port: 8000
    protocol: TCP
  - name: kong-proxy-tls
    port: 8443
    protocol: TCP
#######################################
##        ADD THE 8100 PORT HERE      #
#######################################
  - name: kong-status
    port: 8100
    protocol: TCP

Then save this file and change the serviceMonitor of kong like this ( the port name is the same to the endpoint we added recently ) :

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  generation: 1
  labels:
    prometheus: devops
  name: kong-sm
  namespace: kong
spec:
  endpoints:
  - interval: 30s
#############################################################################
##   THE NAME OF THE PORT IS SAME TO THE NAME WE ADDED TO THE ENDPOINT FILE #
#############################################################################
    port: kong-status
    scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
    - kong
  selector:
    matchLabels:
      app.kubernetes.io/instance: kong
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: kong
      app.kubernetes.io/version: "2.6"
      enable-metrics: "true"
      helm.sh/chart: kong-2.5.0

Apply the serviceMonitor yaml file and after few seconds Prometheus will detect it as a target and scrape kong's metrics successfully.