Monitor Kubernetes Cluster from other Kubernetes cluster with Prometheus

Question

I have run several Kubernetes Clusters on Azure AKS, so I intend to create Centralized Monitoring Cluster run on other Kubernetes cluster with Prometheus-Grafana.

My idea is that:

• Isolating & centralizing monitoring cluster.
• In case of any cluster is downed, Monitoring cluster is still alive to inspect downed cluster.
• Run cross-cloud provider Kubernetes (if available)

I'm confusing about connecting clusters, network, ingress, how does Prometheus discovery, pull metric from outside cluster...

Is there any best practice, instruction for my usecase. Thank you!

Answer 1

Yes, Prometheus is a very flexible monitoring solution wherein each Prometheus server is able to act as a target for another Prometheus server. Using prometheus federation, Prometheus servers can scrape selected time series data from other Prometheus servers.

A typical Prometheus federation example configuration looks like this:

- job_name: 'federate'
  scrape_interval: 15s

  honor_labels: true
  metrics_path: '/federate'

  params:
    'match[]':
      - '{job="prometheus"}'
      - '{__name__=~"job:.*"}'

  static_configs:
    - targets:
      - 'source-prometheus-1:9090'
      - 'source-prometheus-2:9090'
      - 'source-prometheus-3:9090'