Prometheus create label from metric label

Question

We are running the node-exporter in containers. To quickly identify on which host each node-exporter is running, I created a metric that looks like this: host{host="$HOSTNAME",node="$CONTAINER_ID"} 1

I'm looking for a way to extract the hostname in host= and create a label for each node-exporter instance as a hostname label. I tried numerous configurations and none seem to work. Current prometheus config looks like this:

scrape_configs:
   - job_name: 'node'
     scrape_interval: 10s
     scrape_timeout: 5s
     metrics_path: /metrics
     scheme: http
     dns_sd_configs:
     - names:
     - tasks.master-nodeexporter
       refresh_interval: 30s
       type: A
       port: 9100
     relabel_configs:
     - source_labels: ['host']
       regex: '"(.*)".*'
       target_label: 'hostname'
       replacement: '$1'

Answer 1

This is not possible, as target relabelling happens before the scrape.

What you want to do here is use service discovery to have the right hostname in the first place, which is not possible with dns_sd_configs . You might look at something like Consul andhttps://www.robustperception.io/controlling-the-instance-label/

Answer 2

If someone comes across this:

Create this as docker-entrypoint.sh and make it executable.

#!/bin/sh -e
# Must be executable by others

NODE_NAME=$(cat /etc/nodename) echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom

set -- /bin/node_exporter "$@"

exec "$@"

Than create a Dockerfile like this

FROM prom/node-exporter:latest

ENV NODE_ID=none

USER root

COPY conf /etc/node-exporter/

ENTRYPOINT [ "/etc/node-exporter/docker-entrypoint.sh" ]
CMD [ "/bin/node_exporter" ]

Than build it and you will always get the Hostname as a node_meta metric

Answer 3

This answer explains how to export node name via node_name label at node_meta metric. Then it is possible to add node_name label to any metric exposed by node_exporter with group_left() modifier during querying. For example, the following PromQL query adds node_name label from node_meta metric to node_memory_Active_bytes metric:

node_memory_Active_bytes
  * on(job,instance) group_left(node_name)
node_meta

See more details about group_left() modifier in these docs and this article .