FluentBit 無法連接 K8s 節點上運行的所有 pod 的堆棧跟蹤日志，它僅適用於節點上的單個 pod
[英]FluentBit unable to concatenate stack-trace logs for all the pods running on K8s node, it works only for a single pod on the node
問題描述
我們一直在嘗試為 k8s 集群上的日志聚合配置 fluent bit。 我們正在使用 newRelic bundle helm 圖表來實現這一點。 newRelic 包在每個 K8s 集群節點上創建一個 pod，並根據定義的配置處理日志。 除了堆棧跟蹤連接外，一切似乎都運行良好。 問題如下：
我們在單個節點上運行 4 個 pod，運行時它們在“/var/log/containers”目錄下創建以下日志文件。
myapp-svc1-<pod-id>.log
myapp-svc2-<pod-id>.log
myapp-svc3-<pod-id>.log
myapp-svc4-<pod-id>.log
這是配置：
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level ${LOG_LEVEL}
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
[INPUT]
Name tail
Tag kube.*
Path ${PATH}
Parser ${LOG_PARSER}
DB ${FB_DB}
Mem_Buf_Limit 7MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name multiline
Match *
multiline.key_content log
multiline.parser multiline-regex-error-trace
[FILTER]
Name kubernetes
Match kube.*
# We need the full DNS suffix as Windows only supports resolving names with this suffix
# See: https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#dns-limitations
Kube_URL https://kubernetes.default.svc.cluster.local:443
Buffer_Size ${K8S_BUFFER_SIZE}
K8S-Logging.Exclude ${K8S_LOGGING_EXCLUDE}
[FILTER]
Name record_modifier
Match *
Record cluster_name ${CLUSTER_NAME}
Allowlist_key container_name
Allowlist_key namespace_name
Allowlist_key pod_name
Allowlist_key stream
Allowlist_key message
Allowlist_key log
Allowlist_key kubernetes
[OUTPUT]
Name newrelic
Match *
licenseKey ${LICENSE_KEY}
endpoint ${ENDPOINT}
lowDataMode ${LOW_DATA_MODE}
這是我們使用的解析器配置：
parsers.conf: |
[MULTILINE_PARSER]
name multiline-regex-error-trace
type regex
flush_timeout 1000
rule "start_state" "/([0-9]{2,4}\-[0-9]{1,2}\-[0-9]{1,2} [0-9]{1,2}\:[0-9]{1,2}\:[0-9]{1,2}\,[0-9]{2,4}) (.*)/" "stacktraceline2"
rule "stacktraceline2" "/^([a-z]{1,10})\.(.*)/" "stacktraceline3"
rule "stacktraceline3" "/^\s+at.*/" "stacktraceline3"
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
如果我們在 'INPUT' 配置下為 'Path' 鍵提到“/var/log/containers/*.log”，我們是在指示 fluenbit 從該目錄收集所有日志，並進行處理，上帝保佑它完美運行. 但是，在這種情況下，自定義多行解析器“multiline-regex-error-trace”似乎不適用於所有 pod 日志。 日志僅連接其中一個 pod，我們看到每個堆棧跟蹤行分別為所有剩余的 pod 推送。
為了讓這個解析器適用於特定的 pod，我們需要將路徑定義為“/var/log/containers/myapp-svc1-.log ”或“/var/log/containers/myapp-svc2-.log ”，即取決於 pod姓名。
但這不是所需的配置，因為定義 pod 特定路徑將限制僅針對節點上該 pod 的日志收集，並且我們需要所有 pod 的日志。
這是日志示例文件：
2022-09-02 18:46:53,206 ERROR 5d9073b1-9f90-42c1-b7a2-d5a6c13f2669 [http-nio-9002-exec-3] i.f.m.c.c.ContentController: Exception recevied from the service
java.lang.Exception: Custom error
at myapp.content.controller.ContentController.throwError(ContentController.java:46)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at myapp.content.filter.RequestIdAddingFilter.doFilterInternal(RequestIdAddingFilter.java:51)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:764)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:887)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1684)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
我們嘗試使用多行內置 fluentbit 解析器“java”。 但它還有另一個問題。 我們的堆棧跟蹤中有 3 種類型的日志行。
**Line Type 1:** 2022-09-02 18:46:53,206 ERROR 5d9073b1-9f90-42c1-b7a2-d5a6c13f2669 [http-nio-9002-exec-3] i.f.m.c.c.ContentController: Exception recevied from the service
**Line Type 2:** java.lang.Exception: Custom error
**Line Type 3:**
at myapp.content.controller.ContentController.throwError(ContentController.java:46)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
內置的“java”解析器似乎只結合了最后兩種類型“Line Type 2”和“Line Type 3”，它不包括第一行。 newrelic 捆綁包內部使用 fluent bit 1.9.4，這是截至目前的最新版本。 要復制此問題，只需在 k8s 節點上運行 2 個 Pod，並查看每個 Pod 的堆棧跟蹤是否與 fluent bit 1.9.4 連接。 確保如上所述的日志示例。 有人可以幫助我們嗎？過去 7 天我一直在用頭撞牆？
1 個解決方案
解決方案1
0 2022-09-09 11:31:00
它不是放在錯誤的部分嗎？ 在文檔中，他們將multiline.parser添加到INPUT而不是[FILTER]部分。 就像是
[INPUT]
Name tail
Tag kube.*
Path ${PATH}
Parser ${LOG_PARSER}
DB ${FB_DB}
Mem_Buf_Limit 7MB
Skip_Long_Lines On
Refresh_Interval 10
multiline.parser multiline-regex-error-trace
K8s 如何為每個特定節點部署 pod 1，而不是集群中的所有節點
[英]K8s How to deploy pod 1 per specific node, not all node in cluster
如何獲取運行 Node JS 應用程序的 k8s Pod 當前和准確的內存使用情況
[英]How to get Current and accurate Memory Usage of k8s Pod running Node JS application
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.