Replace pod IP address with Host IP address keeping path and port same

Question

Currently I am monitoring Kubernetes Pods using Prometheus. My base config is :

- job_name: 'kubernetes_pods'
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - api_server: http://k8s_master:8080
    role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: pod_name
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)

This basically results the Endpoints column stating the internal Kubernetes IP addresses rather than the host IP addresses on which the pods are deployed.

I want to change the pod IP addresses to the actual host IP addresses.

So I added the below config:

- source_labels: [__meta_kubernetes_pod_host_ip]
  target_label: __address__
  regex: (.*)
  replacement: $1

Now what Im seeing is that the IPs have definitely been changed but for some reason everything after : is exactly the same which is wrong.

Is there something wrong with the regex ?

Answer 1

__address__ includes the port number, so you'll need to either preserve that or take it from another label. For example:

- source_labels: [__meta_kubernetes_pod_host_ip, __address__]
  target_label: __address__
  regex: (.*);.*:(\d+)
  replacement: $1:$2