Prometheus failing to pick up metrics that service discovery can find

Question

I'm trying to scrape with Prometheus Istiod metrics but can see this endpoint in undefined Services so far, not sure what that can be.

The definition of service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"istiod","install.operator.istio.io/owning-resource":"unknown","install.operator.istio.io/owning-resource-namespace":"istio-system","istio":"pilot","istio.io/rev":"default","operator.istio.io/component":"Pilot","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.16.1","release":"istio"},"name":"istiod","namespace":"istio-system"},"spec":{"ports":[{"name":"grpc-xds","port":15010,"protocol":"TCP"},{"name":"https-dns","port":15012,"protocol":"TCP"},{"name":"https-webhook","port":443,"protocol":"TCP","targetPort":15017},{"name":"http-monitoring","port":15014,"protocol":"TCP"}],"selector":{"app":"istiod","istio":"pilot"}}}
  creationTimestamp: "2023-01-22T17:46:10Z"
  labels:
    app: istiod
    install.operator.istio.io/owning-resource: unknown
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio: pilot
    istio.io/rev: default
    operator.istio.io/component: Pilot
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.16.1
    release: istio
  name: istiod
  namespace: istio-system
  resourceVersion: "1477"
  uid: 3bb086a7-ce4c-41db-ab84-a62aa8d32451
spec:
  clusterIP: 10.105.229.189
  clusterIPs:
  - 10.105.229.189
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: grpc-xds
    port: 15010
    protocol: TCP
    targetPort: 15010
  - name: https-dns
    port: 15012
    protocol: TCP
    targetPort: 15012
  - name: https-webhook
    port: 443
    protocol: TCP
    targetPort: 15017
  - name: http-monitoring
    port: 15014
    protocol: TCP
    targetPort: 15014
  selector:
    app: istiod
    istio: pilot
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

The definition of ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: istio-component-monitor
namespace: prometheus
labels:
monitoring: istio-components
release: prom
spec:
jobLabel: istio
targetLabels: [app]
selector:
matchExpressions:
     - {key: istio, operator: In, values: [pilot]}
namespaceSelector:
matchNames: 
istio-system
endpoints:
port: http-monitoring
interval: 15s

This is how it looks in Service Discovery in Prometheus:

__address__="172.17.0.4:15012"
__meta_kubernetes_endpoint_address_target_kind="Pod"
__meta_kubernetes_endpoint_address_target_name="istiod-5b86c45f48-7m2w9"
__meta_kubernetes_endpoint_node_name="istio-prometheus-v1.23.9"
__meta_kubernetes_endpoint_port_name="https-dns"
__meta_kubernetes_endpoint_port_protocol="TCP"
__meta_kubernetes_endpoint_ready="true"
__meta_kubernetes_endpoints_label_app="istiod"
__meta_kubernetes_endpoints_label_install_operator_istio_io_owning_resource="unknown"
__meta_kubernetes_endpoints_label_install_operator_istio_io_owning_resource_namespace="istio-system"
__meta_kubernetes_endpoints_label_istio="pilot"
__meta_kubernetes_endpoints_label_istio_io_rev="default"
__meta_kubernetes_endpoints_label_operator_istio_io_component="Pilot"
__meta_kubernetes_endpoints_label_operator_istio_io_managed="Reconcile"
__meta_kubernetes_endpoints_label_operator_istio_io_version="1.16.1"
__meta_kubernetes_endpoints_label_release="istio"
__meta_kubernetes_endpoints_labelpresent_app="true"
__meta_kubernetes_endpoints_labelpresent_install_operator_istio_io_owning_resource="true"
__meta_kubernetes_endpoints_labelpresent_install_operator_istio_io_owning_resource_namespace="true"
__meta_kubernetes_endpoints_labelpresent_istio="true"
__meta_kubernetes_endpoints_labelpresent_istio_io_rev="true"
__meta_kubernetes_endpoints_labelpresent_operator_istio_io_component="true"
__meta_kubernetes_endpoints_labelpresent_operator_istio_io_managed="true"
__meta_kubernetes_endpoints_labelpresent_operator_istio_io_version="true"
__meta_kubernetes_endpoints_labelpresent_release="true"
__meta_kubernetes_endpoints_name="istiod"
__meta_kubernetes_namespace="istio-system"
__meta_kubernetes_pod_annotation_prometheus_io_port="15014"
__meta_kubernetes_pod_annotation_prometheus_io_scrape="true"
__meta_kubernetes_pod_annotation_sidecar_istio_io_inject="false"
__meta_kubernetes_pod_annotationpresent_prometheus_io_port="true"
__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape="true"
__meta_kubernetes_pod_annotationpresent_sidecar_istio_io_inject="true"
__meta_kubernetes_pod_controller_kind="ReplicaSet"
__meta_kubernetes_pod_controller_name="istiod-5b86c45f48"
__meta_kubernetes_pod_host_ip="192.168.58.2"
__meta_kubernetes_pod_ip="172.17.0.4"
__meta_kubernetes_pod_label_app="istiod"
__meta_kubernetes_pod_label_install_operator_istio_io_owning_resource="unknown"
__meta_kubernetes_pod_label_istio="pilot"
__meta_kubernetes_pod_label_istio_io_rev="default"
__meta_kubernetes_pod_label_operator_istio_io_component="Pilot"
__meta_kubernetes_pod_label_pod_template_hash="5b86c45f48"
__meta_kubernetes_pod_label_sidecar_istio_io_inject="false"
__meta_kubernetes_pod_labelpresent_app="true"
__meta_kubernetes_pod_labelpresent_install_operator_istio_io_owning_resource="true"
__meta_kubernetes_pod_labelpresent_istio="true"
__meta_kubernetes_pod_labelpresent_istio_io_rev="true"
__meta_kubernetes_pod_labelpresent_operator_istio_io_component="true"
__meta_kubernetes_pod_labelpresent_pod_template_hash="true"
__meta_kubernetes_pod_labelpresent_sidecar_istio_io_inject="true"
__meta_kubernetes_pod_name="istiod-5b86c45f48-7m2w9"
__meta_kubernetes_pod_node_name="istio-prometheus-v1.23.9"
__meta_kubernetes_pod_phase="Running"
__meta_kubernetes_pod_ready="true"
__meta_kubernetes_pod_uid="893c8a96-2620-45fd-973a-d0f6d6c19d12"
__meta_kubernetes_service_annotation_kubectl_kubernetes_io_last_applied_configuration="{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"istiod","install.operator.istio.io/owning-resource":"unknown","install.operator.istio.io/owning-resource-namespace":"istio-system","istio":"pilot","istio.io/rev":"default","operator.istio.io/component":"Pilot","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.16.1","release":"istio"},"name":"istiod","namespace":"istio-system"},"spec":{"ports":[{"name":"grpc-xds","port":15010,"protocol":"TCP"},{"name":"https-dns","port":15012,"protocol":"TCP"},{"name":"https-webhook","port":443,"protocol":"TCP","targetPort":15017},{"name":"http-monitoring","port":15014,"protocol":"TCP"}],"selector":{"app":"istiod","istio":"pilot"}}} "
__meta_kubernetes_service_annotationpresent_kubectl_kubernetes_io_last_applied_configuration="true"
__meta_kubernetes_service_label_app="istiod"
__meta_kubernetes_service_label_install_operator_istio_io_owning_resource="unknown"
__meta_kubernetes_service_label_install_operator_istio_io_owning_resource_namespace="istio-system"
__meta_kubernetes_service_label_istio="pilot"
__meta_kubernetes_service_label_istio_io_rev="default"
__meta_kubernetes_service_label_operator_istio_io_component="Pilot"
__meta_kubernetes_service_label_operator_istio_io_managed="Reconcile"
__meta_kubernetes_service_label_operator_istio_io_version="1.16.1"
__meta_kubernetes_service_label_release="istio"
__meta_kubernetes_service_labelpresent_app="true"
__meta_kubernetes_service_labelpresent_install_operator_istio_io_owning_resource="true"
__meta_kubernetes_service_labelpresent_install_operator_istio_io_owning_resource_namespace="true"
__meta_kubernetes_service_labelpresent_istio="true"
__meta_kubernetes_service_labelpresent_istio_io_rev="true"
__meta_kubernetes_service_labelpresent_operator_istio_io_component="true"
__meta_kubernetes_service_labelpresent_operator_istio_io_managed="true"
__meta_kubernetes_service_labelpresent_operator_istio_io_version="true"
__meta_kubernetes_service_labelpresent_release="true"
__meta_kubernetes_service_name="istiod"
__metrics_path__="/metrics"
__scheme__="http"
__scrape_interval__="15s"
__scrape_timeout__="10s"
__tmp_prometheus_job_name

I would expect to see this endpoint being scrapped in Prometheus correctly.

Answer 1

Problem solved: serviceMonitorSelector was an issue in my case: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md