ServiceMonitor Prometheus not working in another namespace

Question

I just installed prometheus operator as indicated here: https://github.com/prometheus-operator/kube-prometheus :

kubectl apply --server-side -f manifests/setup
kubectl wait \
    --for condition=Established \
    --all CustomResourceDefinition \
    --namespace=monitoring
kubectl apply -f manifests/

After that I just tried to setup my own service monitor for grafana as follows:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: in1-grafana-service-monitor
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana
  endpoints:
  - port: http
    interval: 10s

This monitor works just fine and I can see it in the Prometheus /targets and /service-discovery.

The fact is that when I want to create this same service monitor but outside the "monitoring" namespace it just not appears neither in /targets or in /service-discovery. My setup for this service monitor is as follows:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: out1-grafana-service-monitor
  namespace: other-namespace
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana
  namespaceSelector:
    any: true
  endpoints:
  - port: http
    interval: 10s

How can I make Prometheus operator to scrape service monitors (and services) outside the monitoring namespace?

I checked the output of kubectl get prom -Ao yaml and it just displays an empty list:

[...]
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
[...]

Any help will be appreciated.

Thank you.

I expect that the service monitor outside the monitoring namespace works as I need it for other service (Not for Grafana).

Answer 1

Can you maybe try to deploy it using the helm chart in another namespace: https://github.com/prometheus-community/helm-charts/tree/acd6ee3b87886ccd1dcdab1bbec684190a5055fa/charts/kube-prometheus-stack

and set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false ( link ) in the values.

Then check if it works with a serviceMonitor outside of the operator namespace. If it works compare the YAML with your operator and find the difference

Answer 2

After looking at the yaml files I realized that Prometheus doesn't have the permissions to read all namespaces. And after looking at the repository customization examples I found the solution: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/monitoring-additional-namespaces.md

Hope this helps someone else in the future.