servicemonitor doesnot add to prometheus-operator targets

Question

I have deployed a Prometheus-operator on the k8s cluster. Everything works well but I want to monitor MySQL pods that are in another namespace. I create mysqld-exporter pod and svc for it in MariaDB namespace and a servicemonitor for it in the monitoring namespace. I check all the items which are in this link , but this servicemonitor(for mysqld) doesn't add to Prometheus targets. when I change the svc type to nodeport everything works, and metrics are exposed.

 apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: creationTimestamp: "2022-09-11T11:51:46Z" generation: 1 labels: app.kube.netes.io/part-of: kube-prometheus app.kube.netes.io/version: 9.1.2 monitor-app: mysqld-exporter name: mysqld-exporter namespace: monitoring resourceVersion: "2932040" uid: 247683c8-7868-4f2c-9a60-255c703273a5 spec: endpoints: - interval: 30s port: http-metrics jobLabel: k8s-app namespaceSelector: matchNames: - mariadb selector: matchLabels: app: mysqld-exporter -------- apiVersion: v1 kind: Service metadata: creationTimestamp: "2022-09-11T11:50:28Z" labels: app: mysqld-exporter name: mysqld-exporter namespace: mariadb resourceVersion: "2931235" uid: 1b548f89-33a1-4235-b042-8cda5dfc766b spec: clusterIP: 10.109.39.231 clusterIPs: - 10.109.39.231 internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: http-metrics port: 9104 protocol: TCP targetPort: 9104 selector: app: mysqld-exporter sessionAffinity: None type: ClusterIP status: loadBalancer: {}

Answer 1

I checked the Prometheus pod logs, and its error was:

pods is forbidden: User "system:serviceaccount:monitoring:prometheus-k8s" cannot list resource

so I search for this error and find the answer in this link and add pods and services to the resources of Prometheus-k8s ClusterRole.

Answer 2

I had the same issue and modifying the clusterRole permissions as per the above link https://github.com/prometheus-operator/kube-prometheus/issues/483#issuecomment-610427646 solved this issue for me. Now Prometheus can scrape from pods in another namespace.