[英]Collecting logs with fluentbit to loki - Indexing custom labels
I wanted to check the following with you:我想和你一起检查以下内容:
I have this kind of YAML manifest, it is a pod that is triggered when events on my app take place:我有这种 YAML 清单,它是一个在我的应用程序上发生事件时触发的 pod:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2021-11-10T12:13:46Z"
generateName: job-1bb229b1-aa15-4349-b676-ad9cf840b44a-
labels:
app: CalculationPod
calculationId: "1467"
controller-uid: 75d707d6-aab6-446d-9726-7ffafde29191
job-name: job-1bb229b1-aa15-4349-b676-ad9cf840b44a
name: job-1bb229b1-aa15-4349-b676-ad9cf840b44a-h4pj
I am using fluent-bit (from Loki stack) to collect logs in my k8s cluster.我正在使用 fluent-bit(来自 Loki 堆栈)在我的 k8s 集群中收集日志。 Actually, I want to index the
calculationId: "1467"
label I have in the pod, to make it appear in grafana-Loki such as app: CalculationPod
is right now in the picture:实际上,我想对 pod 中的
CalculationPod
ID calculationId: "1467"
标签进行索引,以使其出现在 Grafana-Loki 中,例如 app: CalculationPod
现在在图片中:
So this is my output Loki plugin configuration on fluetbit configmap side:所以这是我在fluetbit configmap 端的输出Loki 插件配置:
I am trying to get the value of calculationId
label and pass it as a parameter to the set of labels at Labels parameter in this way:我正在尝试获取
calculationId
标签的值并将其作为参数以这种方式传递给Labels参数中的标签集:
[Output]
Name grafana-loki
Match *
Url http://loki:3100/loki/api/v1/push
TenantID ""
BatchWait 1
BatchSize 1048576
Labels {job="fluent-bit",calculationId="$calculationId"} # SEE HERE
RemoveKeys kubernetes,stream
AutoKubernetesLabels false
LabelMapPath /fluent-bit/etc/labelmap.json
LineFormat json
LogLevel warn
But I couldn't see nothing has changed in my Loki UI.但是我看不出我的 Loki UI 没有任何变化。 I am not sure if that is the correct way to add new labels to be indexed I am checking this https://docs.fluentbit.io/manual/pipeline/outputs/loki#labels
我不确定这是否是添加要索引的新标签的正确方法我正在检查这个https://docs.fluentbit.io/manual/pipeline/outputs/loki#labels
Any help will be so appreciated.任何帮助将不胜感激。
I managed to get the calculationId
label and its value by adding it to the kubernetes labels JSON information is being referenced and that the kubernetes filter call.我设法通过将它添加到正在引用 JSON 信息和 kubernetes 过滤器调用的 kubernetes 标签中来获取
calculationId
标签及其值。 You can see more about this here你可以在这里看到更多关于这个
So the entire configmap/loki-fluent-bit-loki
configuration file is this:所以整个
configmap/loki-fluent-bit-loki
配置文件是这样的:
apiVersion: v1
data:
fluent-bit.conf: |-
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
Flush 1
Daemon Off
Log_Level warn
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
DB /run/fluent-bit/flb_kube.db
Mem_Buf_Limit 1000MB
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
K8S-Logging.Exclude Off
K8S-Logging.Parser Off
[Output]
Name grafana-loki
Match *
Url http://loki:3100/loki/api/v1/push
TenantID ""
BatchWait 1
BatchSize 1048576
Labels {job="fluent-bit"}
RemoveKeys kubernetes,stream
AutoKubernetesLabels false
LabelMapPath /fluent-bit/etc/labelmap.json
LineFormat json
LogLevel warn
labelmap.json: |-
{
"kubernetes": {
"container_name": "container",
"host": "node",
"labels": {
"app": "app",
"release": "release",
"calculationId": "calculationId" # IT WAS ADDED HERE
},
"namespace_name": "namespace",
"pod_name": "instance"
},
"stream": "stream"
}
parsers.conf: |-
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: loki
meta.helm.sh/release-namespace: monitoring
creationTimestamp: "2021-10-26T10:23:32Z"
labels:
app: fluent-bit-loki
app.kubernetes.io/managed-by: Helm
chart: fluent-bit-2.3.0
heritage: Helm
release: loki
name: loki-fluent-bit-loki
namespace: monitoring
Then I got now the calculationId
label indexed:然后我现在得到了索引的
calculationId
标签:
I didn't try, but according to the way the loki fluentbit output plugin works, the LabelKeys
parameter also allow me to add a custom label like my calculationId
and get it to use it to get its log streams我没有尝试,但根据 loki fluentbit 输出插件的工作方式,
LabelKeys
参数还允许我添加一个自定义标签,如我的calculationId
并让它使用它来获取其日志流
The grafana loki documentation says : grafana loki 文档说:
LabelKeys: Comma separated list of keys to use as stream labels.
LabelKeys:逗号分隔的键列表,用作流标签。 All other keys will be placed into the log line.
所有其他键都将放入日志行中。 LabelKeys is deactivated when using LabelMapPath label mapping configuration.
使用 LabelMapPath 标签映射配置时,LabelKeys 被停用。
So According to this, LabelKeys
should works as well, and then if I use it I wouldn't need to use the LabelMapPath
parameter to reference in json the desired labels.所以根据这个,
LabelKeys
应该可以工作,然后如果我使用它,我就不需要使用LabelMapPath
参数在 json 中引用所需的标签。
IMPORTANT重要的
I am using here the fluent-bit collector agent that comes with the loki stack, it means this installation approach .我在这里使用 loki 堆栈附带的 fluent-bit 收集器代理,这意味着这种安装方法。 Is opportune to highlight this is not the official fluent-bit collector, the official one is this .
适时强调一下,这不是官方的fluent-bit收集器,官方的就是这个。 Right now there is a request to deprecate the fluent-bit grafana loki stack in favor of the official one, so will be opportune to highlight this, and think about to use the official one.
现在有人要求弃用fluent-bit grafana loki 堆栈以支持官方堆栈,因此将适时强调这一点,并考虑使用官方堆栈。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.