How monitoring can be limited to single namespace in prometheus. I am unable to get any namespaces options to set a single namespace
Question
I have used the prometheus deployment from the link "https://raw.githubusercontent.com/istio/istio/release-1.16/samples/addons/prometheus.yaml". We want prometheus to monitor a single namespace which shall help us use prometheus and grafana in single namespace as we do not get to have users seperation based on namespaces in a single deployment for the complete cluster. I did follow the suggestion in Monitor only one namespace metrics - Prometheus with Kube.netes but they have not provided a complete configmap yaml. However with a basic idea I have updated the configmap as below:
```
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
alerting_rules.yml: |
{}
alerts: |
{}
prometheus.yml: |
global:
evaluation_interval: 1m
scrape_interval: 15s
scrape_timeout: 10s
rule_files:
- /etc/config/recording_rules.yml
- /etc/config/alerting_rules.yml
- /etc/config/rules
- /etc/config/alerts
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-apiservers
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: default;kubernetes;https
source_labels:
- __meta_kubernetes_namespace
- __meta_kubernetes_service_name
- __meta_kubernetes_endpoint_port_name
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-nodes
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-nodes-cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
- honor_labels: true
job_name: kubernetes-service-endpoints
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
- action: drop
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
replacement: __param_$1
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: service
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node
- honor_labels: true
job_name: kubernetes-service-endpoints-slow
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
replacement: __param_$1
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: service
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node
scrape_interval: 5m
scrape_timeout: 30s
- honor_labels: true
job_name: prometheus-pushgateway
kubernetes_sd_configs:
- role: service
relabel_configs:
- action: keep
regex: pushgateway
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- honor_labels: true
job_name: kubernetes-services
kubernetes_sd_configs:
- role: service
metrics_path: /probe
params:
module:
- http_2xx
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- source_labels:
- __address__
target_label: __param_target
- replacement: blackbox
target_label: __address__
- source_labels:
- __param_target
target_label: instance
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- source_labels:
- __meta_kubernetes_service_name
target_label: service
- honor_labels: true
job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- canary
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: drop
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
replacement: __param_$1
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: drop
regex: Pending|Succeeded|Failed|Completed
source_labels:
- __meta_kubernetes_pod_phase
- job_name: "kubernetes-cadvisor"
scheme: https
metrics_path: /metrics/cadvisor
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
# disable certificate verification by uncommenting the line below.
#
# insecure_skip_verify: true
authorization:
credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
metric_relabel_configs:
- action: keep
source_labels: [namespace]
regex: canary
- honor_labels: true
job_name: kubernetes-pods-slow
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- canary
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
replacement: __param_$1
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: drop
regex: Pending|Succeeded|Failed|Completed
source_labels:
- __meta_kubernetes_pod_phase
scrape_interval: 5m
scrape_timeout: 30s
recording_rules.yml: |
{}
rules: |
{}
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"alerting_rules.yml":"{}\n","alerts":"{}\n","prometheus.yml":"global:\n evaluation_interval: 1m\n scrape_interval: 15s\n scrape_timeout: 10s\nrule_files:\n- /etc/config/recording_rules.yml\n- /etc/config/alerting_rules.yml\n- /etc/config/rules\n- /etc/config/alerts\nscrape_configs:\n- job_name: prometheus\n static_configs:\n - targets:\n - localhost:9090\n- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\n job_name: kubernetes-apiservers\n kubernetes_sd_configs:\n - role: endpoints\n relabel_configs:\n - action: keep\n regex: default;kubernetes;https\n source_labels:\n - __meta_kubernetes_namespace\n - __meta_kubernetes_service_name\n - __meta_kubernetes_endpoint_port_name\n scheme: https\n tls_config:\n ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n insecure_skip_verify: true\n- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\n job_name: kubernetes-nodes\n kubernetes_sd_configs:\n - role: node\n relabel_configs:\n - action: labelmap\n regex: __meta_kubernetes_node_label_(.+)\n - replacement: kubernetes.default.svc:443\n target_label: __address__\n - regex: (.+)\n replacement: /api/v1/nodes/$1/proxy/metrics\n source_labels:\n - __meta_kubernetes_node_name\n target_label: __metrics_path__\n scheme: https\n tls_config:\n ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n insecure_skip_verify: true\n- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\n job_name: kubernetes-nodes-cadvisor\n kubernetes_sd_configs:\n - role: node\n relabel_configs:\n - action: labelmap\n regex: __meta_kubernetes_node_label_(.+)\n - replacement: kubernetes.default.svc:443\n target_label: __address__\n - regex: (.+)\n replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor\n source_labels:\n - __meta_kubernetes_node_name\n target_label: __metrics_path__\n scheme: https\n tls_config:\n ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n insecure_skip_verify: true\n- honor_labels: true\n job_name: kubernetes-service-endpoints\n kubernetes_sd_configs:\n - role: endpoints\n relabel_configs:\n - action: keep\n regex: true\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_scrape\n - action: drop\n regex: true\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow\n - action: replace\n regex: (https?)\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_scheme\n target_label: __scheme__\n - action: replace\n regex: (.+)\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_path\n target_label: __metrics_path__\n - action: replace\n regex: ([^:]+)(?::\\d+)?;(\\d+)\n replacement: $1:$2\n source_labels:\n - __address__\n - __meta_kubernetes_service_annotation_prometheus_io_port\n target_label: __address__\n - action: labelmap\n regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)\n replacement: __param_$1\n - action: labelmap\n regex: __meta_kubernetes_service_label_(.+)\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_service_name\n target_label: service\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_node_name\n target_label: node\n- honor_labels: true\n job_name: kubernetes-service-endpoints-slow\n kubernetes_sd_configs:\n - role: endpoints\n relabel_configs:\n - action: keep\n regex: true\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow\n - action: replace\n regex: (https?)\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_scheme\n target_label: __scheme__\n - action: replace\n regex: (.+)\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_path\n target_label: __metrics_path__\n - action: replace\n regex: ([^:]+)(?::\\d+)?;(\\d+)\n replacement: $1:$2\n source_labels:\n - __address__\n - __meta_kubernetes_service_annotation_prometheus_io_port\n target_label: __address__\n - action: labelmap\n regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)\n replacement: __param_$1\n - action: labelmap\n regex: __meta_kubernetes_service_label_(.+)\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_service_name\n target_label: service\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_node_name\n target_label: node\n scrape_interval: 5m\n scrape_timeout: 30s\n- honor_labels: true\n job_name: prometheus-pushgateway\n kubernetes_sd_configs:\n - role: service\n relabel_configs:\n - action: keep\n regex: pushgateway\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_probe\n- honor_labels: true\n job_name: kubernetes-services\n kubernetes_sd_configs:\n - role: service\n metrics_path: /probe\n params:\n module:\n - http_2xx\n relabel_configs:\n - action: keep\n regex: true\n source_labels:\n - __meta_kubernetes_service_annotation_prometheus_io_probe\n - source_labels:\n - __address__\n target_label: __param_target\n - replacement: blackbox\n target_label: __address__\n - source_labels:\n - __param_target\n target_label: instance\n - action: labelmap\n regex: __meta_kubernetes_service_label_(.+)\n - source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - source_labels:\n - __meta_kubernetes_service_name\n target_label: service\n- honor_labels: true\n job_name: kubernetes-pods\n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n - action: keep\n regex: true\n source_labels:\n - __meta_kubernetes_pod_annotation_prometheus_io_scrape\n - action: drop\n regex: true\n source_labels:\n - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow\n - action: replace\n regex: (https?)\n source_labels:\n - __meta_kubernetes_pod_annotation_prometheus_io_scheme\n target_label: __scheme__\n - action: replace\n regex: (.+)\n source_labels:\n - __meta_kubernetes_pod_annotation_prometheus_io_path\n target_label: __metrics_path__\n - action: replace\n regex: ([^:]+)(?::\\d+)?;(\\d+)\n replacement: $1:$2\n source_labels:\n - __address__\n - __meta_kubernetes_pod_annotation_prometheus_io_port\n target_label: __address__\n - action: labelmap\n regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)\n replacement: __param_$1\n - action: labelmap\n regex: __meta_kubernetes_pod_label_(.+)\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_name\n target_label: pod\n - action: drop\n regex: Pending|Succeeded|Failed|Completed\n source_labels:\n - __meta_kubernetes_pod_phase\n- honor_labels: true\n job_name: kubernetes-pods-slow\n kubernetes_sd_configs:\n - role: pod\n relabel_configs:\n - action: keep\n regex: true\n source_labels:\n - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow\n - action: replace\n regex: (https?)\n source_labels:\n - __meta_kubernetes_pod_annotation_prometheus_io_scheme\n target_label: __scheme__\n - action: replace\n regex: (.+)\n source_labels:\n - __meta_kubernetes_pod_annotation_prometheus_io_path\n target_label: __metrics_path__\n - action: replace\n regex: ([^:]+)(?::\\d+)?;(\\d+)\n replacement: $1:$2\n source_labels:\n - __address__\n - __meta_kubernetes_pod_annotation_prometheus_io_port\n target_label: __address__\n - action: labelmap\n regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)\n replacement: __param_$1\n - action: labelmap\n regex: __meta_kubernetes_pod_label_(.+)\n - action: replace\n source_labels:\n - __meta_kubernetes_namespace\n target_label: namespace\n - action: replace\n source_labels:\n - __meta_kubernetes_pod_name\n target_label: pod\n - action: drop\n regex: Pending|Succeeded|Failed|Completed\n source_labels:\n - __meta_kubernetes_pod_phase\n scrape_interval: 5m\n scrape_timeout: 30s\n","recording_rules.yml":"{}\n","rules":"{}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"prometheus","chart":"prometheus-15.9.0","component":"server","heritage":"Helm","release":"prometheus"},"name":"prometheus","namespace":"istio-system"}}
creationTimestamp: "2023-02-01T04:57:40Z"
labels:
app: prometheus
chart: prometheus-15.9.0
component: server
heritage: Helm
release: prometheus
name: prometheus
namespace: istio-system
resourceVersion: "10173050"
uid: 2df90740-9fed-4944-8d1a-89f0e7295996
```
The pod is in running state but on grafana I am able to see all namespaces.
To observe just the metric of a single namespace.
1 answers
solution1
0 2023-02-01 23:01:06
If you want monitoring only the current namespace where Prometheus pod runs, then put the following namespaces config option inside every kube.netes_sd_config :
kubernetes_sd_configs:
- ... your config here
namespaces:
own_namespace: true
If you need monitoring only the specified namespace (for example, foobar ), then use the following config:
- ... your config here
namespaces:
names:
- foobar
See kube.netes_sd_configs docs for more details.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Issue with monitoring custom service on prometheus in kubernetes namespace
How to have a single user with access to multiple namespaces with separate certs for each namespace
Installed prometheus-community / helm-charts but I can't get metrics on "default" namespace
How to list k8 namespace in a single line
How can I set the helm tiller namespace via an environment variable?
'kubectl get namespace' command “namespaces is forbidden” error
possible to set up kubernetes monitoring in a different namespace
Can I set a default namespace in Kubernetes?
Monitoring resources per namespace
how to move prometheus adapter to another namespace?
Related Tags