Installed prometheus-community / helm-charts but I can't get metrics on "default" namespace

Question

I recently learned about helm and how easy it is to deploy the whole prometheus stack for monitoring a Kube.netes cluster, so I decided to try it out on a staging cluster at my work.

I started by creating a dedicates namespace on the cluster for monitoring with:

kubectl create namespace monitoring

Then, with helm , I added the prometheus-community repo with:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Next, I installed the chart with a prometheus release name:

helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring

At this time I didn't pass any custom configuration because I'm still trying it out.

After the install is finished, it all looks good. I can access the prometheus dashboard with:

kubectl port-forward prometheus-prometheus-kube-prometheus-prometheus-0 9090 -n monitoring

There, I see a bunch of pre-defined alerts and rules that are monitoring but the problem is that I don't quite understand how to create new rules to check the pods in the default namespace, where I actually have my services deployed.

I am looking at http://localhost:9090/graph to play around with the queries and I can't seem to use any that will give me metrics on my pods in the default namespace.

I am a bit overwhelmed with the amount of information so I would like to know what did I miss or what am I doing wrong here?

Answer 1

The Prometheus Operator includes several Custom Resource Definitions (CRDs) including ServiceMonitor (and PodMonitor ). ServiceMonitor 's are used to define services to the Operator to be monitored.

I'm familiar with the Operator although not the Helm deployment but I suspect you'll want to create ServiceMonitors to generate metrics for your apps in any (including default ) namespace.

See: https://github.com/prometheus-operator/prometheus-operator#customresourcedefinitions

Answer 2

ServiceMonitors and PodMonitors are CRDs for Prometheus Operator. When working directly with Prometheus helm chart (without operator), you need have to configure your targets directly in values.yaml by editing the scrape_configs section.

It is more complex to do it, so take a deep breath and start by reading this: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config