Prometheus node exporter is registered as a service with consul agent with various tags. Example service definition provided to consul agent:
{
"service":{
"id": "server-stats",
"name": "server-stats",
"tags": [
"a=1_meow",
"b=2_woof",
"c=3_moo",
"monkey"
],
"port": 9100,
"checks": [
{
"name": "Process #1",
"script": "/path/to/healthcheck/script.sh",
"interval": "5s"
}
]
}
}
Prometheus is set to look for this server-stats
service and use the configuration (host address and port) provided by Consul to scrape stats from servers. The above tags are available as a comma separated list in __meta_consul_tags
that can be used for relabeling.
Prometheus relabeling configuration:
relabel_configs:
- source_labels: [__meta_consul_tags]
separator: ','
#regex: '(.+)=(.+)'
regex: '([a-z_]+)=([a-z_]+|\d+)'
target_label: ${1}
replacement: ${2}
I am trying to expose tags to Prometheus so that we can get stats and graphs based on labels. Keeping the above service configuration in mind, I would like each metric to have following labels in addition to whatever Prometheus does internally: a=1_meow
, b=2_woof
, c=3_moo
and ignore monkey
because it is just a string. I can remove monkey
from my list of tags if there is a solution that requires =
to be there. The relabel configuration written above is not leading to exposing any tag at all and seems to be getting ignored. Running Prometheus with log level set to debug is also not yielding anything.
I think there was a mistake in my understanding of how labeling in prometheus works. My incorrect understanding was:
regex
, string would be first split on separator
(otherwise what is its purpose?), regex
evaluated against it, target_label
and replacement
fields. regex
does not match, then that substring will be ignored. regex
is expected to be applied to each substring after the split, it will lead to multiple labels from multiple substrings. However, from brian-brazil 's post linked in his answer and Prometheus's documentation, it seems the following happening:
__meta
tags are combined into one long separator
separated line. regex
is applied on that line only once. regex
matches and includes groups, they are indexed beginning from 1 and available for use in target_label
and replacement
. separator
seems to be getting ignored in this section even if you mention it. From this idea and following from example in the question, I was able to make the following config that works
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,a=([a-z0-9_]+),.+'
target_label: 'a'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,b=([a-z0-9_]+),.+'
target_label: 'b'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,c=([a-z0-9_]+),.+'
target_label: 'c'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,d=([a-z0-9_]+),.+'
target_label: 'd'
replacement: ${1}
I believe both approaches (the approach brian-brazil wrote in his blogpost , and what I am using above) have caveats - we either need to know all the labels we want beforehand, or have a set number of them. This means if a developer wants to associate different, or more labels with his/her service, s/he would need to work with ops as general flow will not be able to handle it. I think it is a minor caveat that should be addressed.
https://www.robustperception.io/extracting-full-labels-from-consul-tags/显示了如何执行此操作,特别是最后一个示例。
The following relabeling rule can be used for extracting a particular tag
from __meta_consul_tags
and placing it to destination_label
:
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,tag=([^,]+),.*'
target_label: destination_label
There is no need to specify the replacement
option because it defaults to $1
.
If multiple tags must be extracted from __meta_consul_tags
into multiple labels, then just repeat the relabeling rule per each needed tag. For example, the following relabeling rules extract a
tag to a
label and b
tag to b
label:
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,a=([^,]+),.*'
target_label: a
- source_labels: [__meta_consul_tags]
regex: '.*,b=([^,]+),.*'
target_label: b
More information about relabeling rules in Prometheus is available in this article .
Update (2022-12-19): VictoriaMetrics and vmagent (these are Prometheus-like monitoring systems and scrapers I work on) expose __meta_consul_tag_<tagname>
and __meta_consul_tagpresent_<tagname>
labels for discovered targets starting from v1.85.2 . This allows copying all the tags defined at Consul service into the scrape target labels with a single relabeling rule:
- action: labelmap
regex: __meta_consul_tag_(.+)
Try playing with this relabeling rule at Prometheus relabel debugger .
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.