How to add new labels to existing prometheus metric in Istio?

Question

I'm trying to add a new label source_ip to the prometheus metric requestcount
I've added the attribute to prom handler

params:
    metrics:
    - instance_name: requestcount.instance.istio-system
      kind: COUNTER
      label_names:
      - reporter
      - source_ip
      - source_app

and added a dimension to requestcount instance

compiledTemplate: metric
  params:
    dimensions:
      reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
      source_app: source.labels["app"] | "unknown"
      source_ip: source.ip | "unknown"

and added an attribute_binding to the attributes instance

spec:
  attributeBindings:
    destination.workload.namespace: $out.destination_workload_namespace | "unknown"
    destination.workload.uid: $out.destination_workload_uid | "unknown"
    source.ip: $out.source_pod_ip | ip("0.0.0.0")

yet, source_ip label is not included in the istio_request_total metric reported by prometheus, am I missing something here?

Answer 1

About mixer and documentation you use

I'm using istio 1.5 and upgrading might take me some considerable time.

That documentation you mentioned won't work on istio 1.5 as it uses mixer which is deprecated since istio 1.5, as mentioned in below docs you might re-enable it but I couldn't find any documentation about that.

As mentioned here and here

Mixer is deprecated. The functionality provided by Mixer is being moved into the Envoy proxies. Use of Mixer with Istio will only be supported through the 1.7 release of Istio.

Mixer deprecation

Mixer, the process behind the istio-telemetry and istio-policy deployments, has been deprecated with the 1.5 release. istio-policy was disabled by default since Istio 1.3 and istio-telemetry is disabled by default in Istio 1.5.

Telemetry is collected using an in-proxy extension mechanism (Telemetry V2) that does not require Mixer.

If you depend on specific Mixer features like out of process adapters, you may re-enable Mixer. Mixer will continue receiving bug fixes and security fixes until Istio 1.7. Many features supported by Mixer have alternatives as specified in the Mixer Deprecation document including the in-proxy extensions based on the WebAssembly sandbox API.

If you rely on a Mixer feature that does not have an equivalent, we encourage you to open issues and discuss in the community.

---

About upgrade

About the upgrade, I would say if that was older version of istio then it might be harder to upgrade, but since it's 1.5 I would say it might be easy to upgrade to 1.6 with istioctl upgrade . I would suggest to test it first on some test environment.

About the main question

Istio configures prometheus with a 'kubernetes-pods' job. At least while using the 'demo' profile. In this prometheus job config, there is a relabel_configs which gets the pod labels.

relabel_configs: 
...
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)

If you want to use it then use meshConfig.enablePrometheusMerge=true option, it will append the labels to the istio metrics. There is related documentation about that. Just a notice that this option is newly introduced in Istio 1.6 and is considered alpha at this time.