[英]Relabeling in Prometheus
Prometheus node exporter is registered as a service with consul agent with various tags. Prometheus node exporter 注册为具有各种标签的 consul agent 的服务。 Example service definition provided to consul agent:
提供给领事代理的示例服务定义:
{
"service":{
"id": "server-stats",
"name": "server-stats",
"tags": [
"a=1_meow",
"b=2_woof",
"c=3_moo",
"monkey"
],
"port": 9100,
"checks": [
{
"name": "Process #1",
"script": "/path/to/healthcheck/script.sh",
"interval": "5s"
}
]
}
}
Prometheus is set to look for this server-stats
service and use the configuration (host address and port) provided by Consul to scrape stats from servers. Prometheus 被设置为寻找这个
server-stats
服务并使用 Consul 提供的配置(主机地址和端口)从服务器中抓取统计信息。 The above tags are available as a comma separated list in __meta_consul_tags
that can be used for relabeling.上述标签在
__meta_consul_tags
中以逗号分隔的列表形式提供,可用于重新标记。
Prometheus relabeling configuration:普罗米修斯重新标记配置:
relabel_configs:
- source_labels: [__meta_consul_tags]
separator: ','
#regex: '(.+)=(.+)'
regex: '([a-z_]+)=([a-z_]+|\d+)'
target_label: ${1}
replacement: ${2}
I am trying to expose tags to Prometheus so that we can get stats and graphs based on labels.我正在尝试将标签公开给普罗米修斯,以便我们可以根据标签获取统计数据和图表。 Keeping the above service configuration in mind, I would like each metric to have following labels in addition to whatever Prometheus does internally:
a=1_meow
, b=2_woof
, c=3_moo
and ignore monkey
because it is just a string.牢记上述服务配置,除了 Prometheus 内部所做的任何事情之外,我希望每个指标都有以下标签:
a=1_meow
、 b=2_woof
、 c=3_moo
并忽略monkey
,因为它只是一个字符串。 I can remove monkey
from my list of tags if there is a solution that requires =
to be there.如果有需要
=
的解决方案,我可以从我的标签列表中删除monkey
。 The relabel configuration written above is not leading to exposing any tag at all and seems to be getting ignored.上面写的 relabel 配置根本不会导致暴露任何标签,而且似乎被忽略了。 Running Prometheus with log level set to debug is also not yielding anything.
在日志级别设置为调试的情况下运行 Prometheus 也不会产生任何结果。
I think there was a mistake in my understanding of how labeling in prometheus works. 我认为我对普罗米修斯的标签如何运作有一个错误。 My incorrect understanding was:
我的错误理解是:
regex
, string would be first split on separator
(otherwise what is its purpose?), regex
之前,首先将字符串拆分为separator
(否则它的目的是什么?), regex
evaluated against it, regex
, target_label
and replacement
fields. target_label
和replacement
字段的索引值。 regex
does not match, then that substring will be ignored. regex
不匹配,则该子字符串将被忽略。 regex
is expected to be applied to each substring after the split, it will lead to multiple labels from multiple substrings. regex
应用于每个子字符串,所以它将导致来自多个子字符串的多个标签。 However, from brian-brazil 's post linked in his answer and Prometheus's documentation, it seems the following happening: 然而,从他的答案和普罗米修斯的文档中链接的布里安 - 巴西的帖子看来,似乎发生了以下情况:
__meta
tags are combined into one long separator
separated line. __meta
标签组合成一个长separator
分隔线。 regex
is applied on that line only once. regex
仅在该行上应用一次。 regex
matches and includes groups, they are indexed beginning from 1 and available for use in target_label
and replacement
. regex
匹配并包含组,则它们从1开始编制索引,并可用于target_label
和replacement
。 separator
seems to be getting ignored in this section even if you mention it. separator
似乎也会在本节中被忽略。 From this idea and following from example in the question, I was able to make the following config that works 从这个想法和问题中的例子,我能够使以下配置工作
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,a=([a-z0-9_]+),.+'
target_label: 'a'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,b=([a-z0-9_]+),.+'
target_label: 'b'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,c=([a-z0-9_]+),.+'
target_label: 'c'
replacement: ${1}
- source_labels: [__meta_consul_tags]
regex: '.*,d=([a-z0-9_]+),.+'
target_label: 'd'
replacement: ${1}
I believe both approaches (the approach brian-brazil wrote in his blogpost , and what I am using above) have caveats - we either need to know all the labels we want beforehand, or have a set number of them. 我相信这两种方法(brian-brazil在他的博客中写的方法,以及我上面使用的方法)都有警告 - 我们要么事先知道我们想要的所有标签,要么有一定数量的标签。 This means if a developer wants to associate different, or more labels with his/her service, s/he would need to work with ops as general flow will not be able to handle it.
这意味着如果开发人员想要将不同或更多标签与他/她的服务相关联,他/她将需要使用操作,因为一般流程将无法处理它。 I think it is a minor caveat that should be addressed.
我认为这是一个应该解决的小问题。
https://www.robustperception.io/extracting-full-labels-from-consul-tags/显示了如何执行此操作,特别是最后一个示例。
The following relabeling rule can be used for extracting a particular tag
from __meta_consul_tags
and placing it to destination_label
:以下重新标记规则可用于从
__meta_consul_tags
中提取特定tag
并将其放置到destination_label
:
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,tag=([^,]+),.*'
target_label: destination_label
There is no need to specify the replacement
option because it defaults to $1
.无需指定
replacement
选项,因为它默认为$1
。
If multiple tags must be extracted from __meta_consul_tags
into multiple labels, then just repeat the relabeling rule per each needed tag.如果必须将多个标签从
__meta_consul_tags
中提取到多个标签中,则只需为每个需要的标签重复重新标记规则。 For example, the following relabeling rules extract a
tag to a
label and b
tag to b
label:例如,以下重新标记规则将
a
标签提取到a
标签,将b
标签提取到b
标签:
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: '.*,a=([^,]+),.*'
target_label: a
- source_labels: [__meta_consul_tags]
regex: '.*,b=([^,]+),.*'
target_label: b
More information about relabeling rules in Prometheus is available in this article . 本文提供了有关 Prometheus 中重新标记规则的更多信息。
Update (2022-12-19): VictoriaMetrics and vmagent (these are Prometheus-like monitoring systems and scrapers I work on) expose __meta_consul_tag_<tagname>
and __meta_consul_tagpresent_<tagname>
labels for discovered targets starting from v1.85.2 .更新(2022-12-19):VictoriaMetrics 和vmagent (这些是我使用的类似 Prometheus 的监控系统和爬虫)从v1.85.2开始为发现的目标公开
__meta_consul_tag_<tagname>
和__meta_consul_tagpresent_<tagname>
标签。 This allows copying all the tags defined at Consul service into the scrape target labels with a single relabeling rule:这允许使用单个重新标记规则将 Consul 服务中定义的所有标签复制到抓取目标标签中:
- action: labelmap
regex: __meta_consul_tag_(.+)
Try playing with this relabeling rule at Prometheus relabel debugger .尝试在Prometheus relabel debugger中使用这个重新标记规则。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.