[英]Why does EKS say my fluent-bit.conf is not valid
I am trying to setup Fluent Bit for Kuberentes on EKS + Fargate.我正在尝试在 EKS + Fargate 上为 Kuberentes 设置 Fluent Bit。 I was able to get logs all going to one general log group on Cloudwatch but now when I add fluent-bit.conf: |
我能够将日志全部转到 Cloudwatch 上的一个通用日志组,但现在当我添加 fluent-bit.conf 时:| to the data: field and try to apply the update to my cluster, I get this error:
到数据:字段并尝试将更新应用到我的集群,我收到此错误:
for: "fluentbit-config.yaml": admission webhook "0500-amazon-eks-fargate-configmaps-admission.amazonaws.com" denied the request: fluent-bit.conf is not valid.
对于:“fluentbit-config.yaml”:admission webhook“0500-amazon-eks-fargate-configmaps-admission.amazonaws.com”拒绝请求:fluent-bit.conf 无效。 Please only provide output.conf, filters.conf or parsers.conf in the logging configmap
请仅在日志配置映射中提供 output.conf、filters.conf 或 parsers.conf
What sticks out the most to me is that the error message is asking me to only provide output, filter or parser configurations.对我来说最突出的是错误消息要求我仅提供 output、过滤器或解析器配置。
It matches up with other examples I found online, but it seems like I do not have the fluent-bit.conf file on the cluster that I am updating or something.它与我在网上找到的其他示例相匹配,但似乎我正在更新的集群上没有 fluent-bit.conf 文件。 The tutorials I have followed do not mention installing a file so I am lost as to why I am getting this error.
我遵循的教程没有提到安装文件,所以我不知道为什么会收到此错误。
The这
My fluentbit-config.yaml file looks like this我的 fluentbit-config.yaml 文件看起来像这样
kind: Namespace
apiVersion: v1
metadata:
name: aws-observability
labels:
aws-observability: enabled
---
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
@INCLUDE input-kubernetes.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Parser docker
Tag logger
Path /var/log/containers/*logger-server*.log
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match logger
region us-east-1
log_group_name fluent-bit-cloudwatch
log_stream_prefix from-fluent-bit-
auto_create_group On
As per docs (at the very bottom of that page and yeah, we're in the process of improving them, not happy with the current state) you have a couple of sections in there that are not allowed in the context of EKS on Fargate logging, more specifically what can go into the ConfigMap
.根据文档(在该页面的最底部,是的,我们正在改进它们,对当前状态不满意)你有几个部分在 Fargate 上的 EKS 上下文中是不允许的日志记录,更具体地说,可以将 go 放入
ConfigMap
中。 What you want is something along the lines of the following (note: this is from an actual deployment I'm using, slightly adapted):您想要的是以下内容(注意:这是来自我正在使用的实际部署,稍作修改):
kind: ConfigMap
apiVersion: v1
metadata:
name: aws-logging
namespace: aws-observability
data:
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match *
region eu-west-1
log_group_name something-fluentbit
log_stream_prefix fargate-
auto_create_group On
[OUTPUT]
Name es
Match *
Host blahblahblah.eu-west-1.es.amazonaws.com
Port 443
Index something
Type something_type
AWS_Auth On
AWS_Region eu-west-1
tls On
With this config, you're streaming logs to both CW and AES, so feel free to drop the second OUTPUT section if not needed.使用此配置,您可以将日志流式传输到 CW 和 AES,因此如果不需要,请随意删除第二个 OUTPUT 部分。 However, you notice that there can not be the other sections that you had there such as
input-kube.netes.conf
for example.但是,您注意到那里不能有其他部分,例如
input-kube.netes.conf
。
I wonder if anyone managed to process the 'log' section with fargate 'hide-car' using parser as per fluentbit conf documentation .我想知道是否有人按照 fluentbit conf 文档使用解析器设法使用 fargate 'hide-car' 处理 'log' 部分。 Here's a snippet of my aws-logging config map which pushes logs to both outputs but sadly the parsing is never happening.
这是我的 aws-logging 配置 map 的一个片段,它将日志推送到两个输出,但遗憾的是解析从未发生。
I would like to avoid using hacky regexes when viewing logs in Opensearch which can be avoided with proper parsing of the 'logs'.在 Opensearch 中查看日志时,我想避免使用 hacky 正则表达式,这可以通过正确解析“日志”来避免。
PS.附言。 I noticed fluentbit docs refer to so called 'docker' parser but fargate nodes are using containerd as the container runtime which could potentially be a problem?
我注意到 fluentbit 文档提到了所谓的“docker”解析器,但 fargate 节点正在使用 containerd 作为容器运行时,这可能是一个问题?
data:
filters.conf: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Merge_Log_Key log_proccessed
Buffer_Size 0
Kube_Meta_Cache_TTL 300s
Parser docker
flb_log_cw: 'true'
output.conf: |
[OUTPUT]
Name cloudwatch_logs
Match *
region eu-west-1
log_group_name /aws/eks/bs-277-main/container
log_stream_prefix log-
[OUTPUT]
Name es
Match *
Host vpc-my-amazing-os-endpoint.eu-west-1.es.amazonaws.com
Port 443
Index kubernetes
Type doc
AWS_Auth On
AWS_Region eu-west-1
tls On
parsers.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
Came across this example of fluentbit config with containerd log parsing but it is based on adding Parser param to [INPUT] section which is ignored in Fargate as it is presumably managed by AWS.遇到了这个使用 containerd 日志解析的 fluentbit 配置示例,但它基于将 Parser 参数添加到 [INPUT] 部分,这在 Fargate 中被忽略,因为它可能由 AWS 管理。
It is very unfortunate that crucial component of observability such as fluentbit has so little documentation on AWS Fargate.非常不幸的是,可观察性的关键组件(例如 fluentbit)在 AWS Fargate 上的文档太少了。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.