prometheus operator is not scraping all pods

Question

I have deployed prometheus-operator on digitalocean cluster. using kube-prometheus-stack https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack . I have added some additional scrape config for kubernetes pod role.

- job_name: kubernetes-pods
      scrape_timeout: 5m
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: kubernetes_pod_name
      - action: drop
        regex: Pending|Succeeded|Failed
        source_labels:
        - __meta_kubernetes_pod_phase

after that i have deployed postgres db on postgres namesapce by giving prometheus annotation in the postgres deploy yaml file, here is my file

kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "postgres"
    prometheus.io/path: "/metrics"
    prometheus.io/probe: "true"
  namespace: postgres
  name: postgres
  labels:
    component: postgres
spec:

  ports:
    - port: 5432
      name: postgres
  selector:
    component: postgres
---
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: postgres
  labels:
    component: postgres
spec:
  selector:
    matchLabels:
      component: postgres
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "postgres"
        prometheus.io/path: "/metrics"
      labels:
        component: postgres
    spec:
      containers:
      - image: postgres:11.1-alpine
        name: postgres
        securityContext:
          runAsUser: 70 # postgres user on Alpine
          allowPrivilegeEscalation: false
        resources:
          limits:
            memory: 2Gi
            cpu: 2
          requests:
            memory: 2Gi
            cpu: 2
        env:
          - name: PGDATA
            value: "/var/lib/postgresql/data/pgdata"
          - name: POSTGRES_DB
            valueFrom:
              secretKeyRef:
                name: postgres-secret
                key: PG_DATABASE
          - name: POSTGRES_USER
            valueFrom:
              secretKeyRef:
                name: postgres-secret
                key: PG_USERNAME
          - name: POSTGRES_PASSWORD
            valueFrom:
              secretKeyRef:
                name: postgres-secret
                key: PG_PASSWORD
        ports:
        - containerPort: 5432
          name: postgres
        volumeMounts:
          - name: postgres-storage
            mountPath: /var/lib/postgresql/data
        readinessProbe:
          tcpSocket:
            port: 5432
          initialDelaySeconds: 15
          periodSeconds: 5
        livenessProbe:
          exec:
            command:
            - pg_isready
            - -h
            - localhost
            - -U
            - test_user
            - -d
            - test_db
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
        - name: postgres-storage
          emptyDir: {}

but in prometheus targets for postgres pod i am getting EOF error, can some one help me with this issue?

Answer 1

You need to use an exporter to export metrics in prometheus format.

You can't just send an HTTP request to postgres port hoping that it will give you some metrics.

Use postgress exporter . It talks with postgres and exposes metrics under http endpoint. And what's more important: in format that promethus understands.