Prometheus adapter - not able to get data from external metrics

Question

I am not able to fetch items by running this command: kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/rabbitmq_queue_messages_ready" | jq kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/rabbitmq_queue_messages_ready" | jq

As you can see in below output items is an empty array:

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/rabbitmq_queue_messages_ready"
  },
  "items": []
}

However, I am getting proper data in prometheus endpoint: Prometheus URL: http://<prometheus_url>:9090/api/v1/series?match%5B%5D=%7B__name__%3D~%22%5Erabbitmq_queue_.%2A%22%7D&start=1597255421.51 Response:

{
"status":"success",
"data":[
    {
        "__name__":"rabbitmq_queue_messages_ready",
        "app":"prometheus-rabbitmq-exporter",
        "durable":"true",
        "instance":"10.2.0.73:9419",
        "job":"kubernetes-pods",
        "namespace":"default",
        "pod_name":"rabbitmq-exporter-prometheus-rabbitmq-exporter-754c845847-gzlrq",
        "pod_template_hash":"754c845847",
        "queue":"test",
        "release":"rabbitmq-exporter",
        "vhost":"/"
    },
    {
        "__name__":"rabbitmq_queue_messages_ready",
        "app":"prometheus-rabbitmq-exporter",
        "durable":"true",
        "instance":"10.2.0.73:9419",
        "job":"kubernetes-pods",
        "namespace":"default",
        "pod_name":"rabbitmq-exporter-prometheus-rabbitmq-exporter-754c845847-gzlrq",
        "pod_template_hash":"754c845847",
        "queue":"test1",
        "release":"rabbitmq-exporter",
        "vhost":"/"
    }
]
}

I installed stable/prometheus-adapter using below helm values:

rules:
  default: false
  external:
    - seriesQuery: '{__name__=~"^rabbitmq_queue_.*"}'
      resources:
        #template: <<.Resource>>
        overrides:
          namespace:
            resource: namespace
          service:
            resource: service
          pod:
            resource: pod
      name:
        matches: ""
        as: "rabbitmq_queue_messages_ready"
      metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[1m])'

Version of Helm and Kubernetes :

Client: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"ec831747a3a5896dbdf53f259eafea2a2595217c", GitTreeState:"clean", BuildDate:"2020-05-29T19:56:10Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

Expected result : It should fetch metrics from Prometheus

How to reproduce it : Install stable/prometheus-rabbitmq-exporter and stable/prometheus-adapter helm charts using above configs.

I am following this guide: https://nuvalence.io/building-a-k8s-autoscaler-with-custom-metrics/

Answer 1

To troubleshoot your issue further at first place I would check a couple of things in your prom-adapter setup:

1. Do you see any errors in Prometheus adapter logs while getting metrics directly with "kubectl get --raw..." command?
2. To which namespace you have installed your prometheus-adapter ? (asking because I used to observe RBAC related errors in adapter logs when deployed to another ns than 'kube-system')
3. Can you please paste the content of your final 'prom-adapter' /etc/adapter/config.yaml config (from prom-adapter's Pod)

I'm exposing the same external rabbitmq metrics with the adapter without any problems, check output from the same command:

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/rabbitmq_queue_messages_ready"
  },
  "items": [
    {
      "metricName": "rabbitmq_queue_messages_ready",
      "metricLabels": {
        "endpoint": "metrics",
        "instance": "10.64.1.97:9419",
        "job": "rabbitmq",
        "namespace": "default",
        "pod": "rabbitmq-0",
        "service": "rabbitmq"
      },
      "timestamp": "2020-08-13T23:50:29Z",
      "value": "174160m"
    }
  ]
}

Used prom-adapter version : directxman12/k8s-prometheus-adapter-amd64:v0.6.0

//Using Community wiki post for better clarity (code snippets formatting, etc.)