EKS - Fluent-bit，CloudWatch 無法從日志條目中刪除 Kube.netes 數據

Question

我們已將 Fluent-bit 配置為將日志從我們的集群直接發送到 CloudWatch。 我們啟用了 Kube.netes 過濾器，以便將我們的 log_stream_name 設置為 $(kube.netes['container_name'])。

但是，日志很糟糕。

每條 CloudWatch 行如下所示：

    2022-06-23T14:17:34.879+02:00   {"kubernetes":{"redacted_redacted":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25","redacted_image":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45","redacted_name":"redacted-redacted","docker_id":"b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a","host":"ip-0.0.0.0.region-#.compute.internal","namespace_name":"namespace","pod_id":"podpodpod-296c-podpod-8954-podpodpod","pod_name":"redacted-redacted-redacted-7dcbfd4969-mb5f5"},
    2022-06-23T14:17:34.879+02:00   {"kubernetes":{"redacted_redacted":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25","redacted_image":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45","redacted_name":"redacted-redacted","docker_id":"b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a","host":"ip-0.0.0.0.region-#.compute.internal","namespace_name":"namespace","pod_id":"podpodpod-296c-podpod-8954-podpodpod","pod_name":"redacted-redacted-redacted-7dcbfd4969-mb5f5"},
    2022-06-23T14:17:34.879+02:00   {"kubernetes":{"redacted_redacted":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25","redacted_image":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45","redacted_name":"redacted-redacted","docker_id":"b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a","host":"ip-0.0.0.0.region-#.compute.internal","namespace_name":"namespace","pod_id":"podpodpod-296c-podpod-8954-podpodpod","pod_name":"redacted-redacted-redacted-7dcbfd4969-mb5f5"},
    2022-06-23T14:20:07.074+02:00   {"kubernetes":{"redacted_redacted":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25","redacted_image":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45","redacted_name":"redacted-redacted","docker_id":"b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a","host":"ip-0.0.0.0.region-#.compute.internal","namespace_name":"namespace","pod_id":"podpodpod-296c-podpod-8954-podpodpod","pod_name":"redacted-redacted-redacted-7dcbfd4969-mb5f5"},

這使得日志無法使用，除非展開，一旦展開，日志如下所示：

2022-06-23T14:21:34.207+02:00
{
    "kubernetes": {
        "container_hash": "145236632541.lfl.ecr.region.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25",
        "container_image": "145236632541.lfl.ecr.region-#.amazonaws.com/redacted:ve3b56a45",
        "container_name": "redacted-redacted",
        "docker_id": "b431f9788f46sd5f4ds65f4sd56f4sd65f4d336fff4ca8030a216ecb9e0a",
        "host": "ip-0.0.0.0.region-#.compute.internal",
        "namespace_name": "redacted",
        "pod_id": "podpodpod-296c-podpod-8954-podpodpod",
        "pod_name": "redacted-redacted-redacted-7dcbfd4969-mb5f5"
    },
    "log": "[23/06/2022 12:21:34] loglineloglinelogline\ loglineloglinelogline \n",
    "stream": "stdout"
}
    {"kubernetes":{"redacted_redacted":"145236632541.lfl.ecr.region-#.amazonaws.com/redacted@sha256:59392fab7hsfghsfghsfghsfghsfghsfghc39c1bee75c0b4bfc2d9f4a405aef449b25","redacted_image

這也有點可怕，因為每一行都充斥着 Kube.netes 數據。 我想從日志中完全刪除 Kube.netes 數據，但我想繼續使用 $(kube.netes['container_name']) 作為日志 stream 名稱，以便正確命名日志。 我嘗試使用帶有 Remove_key 和 LUA 腳本的過濾器來刪除 Kube.netes 數據。 但是一旦刪除它，日志 stream 就不能命名為 $(kube.netes['container_name'])。

我發現這方面的文檔很少。 並且還沒有找到刪除 Kube.netes 數據並將我的 log_stream_name 保留為我的 container_name 的正確方法。

這是我使用的流利位配置的原始文件： https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset /container-insights-monitoring/fluent-bit/fluent-bit-compatible.yaml

任何幫助，將不勝感激。

Answer 1

有一條指令https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html /（可選）從 Fluent Bit 減少日志量

只需在日志配置中添加嵌套過濾器。 例如

user-api.conf: |
[INPUT]
    Name                tail
    Tag                 user-api.*
    Path                /var/log/containers/user-api*.log
    Docker_Mode         On
    Docker_Mode_Flush   5
    Docker_Mode_Parser  container_firstline_user
    Parser              docker
    DB                  /var/fluent-bit/state/flb_user_api.db
    Mem_Buf_Limit       50MB
    Skip_Long_Lines     On
    Refresh_Interval    10
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
    Name                kubernetes
    Match               user-api.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     user-api.var.log.containers.
    Merge_Log           On
    Merge_Log_Key       log_processed
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              Off
    Annotations         Off

[FILTER]
    Name                grep
    Match               user-api.*
    Exclude             log /.*"GET \/ping HTTP\/1.1" 200.*/
    
[FILTER]
    Name                nest
    Match               user-api.*
    Operation           lift
    Nested_under        kubernetes
    Add_prefix          Kube.

[FILTER]
    Name                modify
    Match               user-api.*
    Remove              kubernetes.kubernetes.host
    Remove              Kube.container_hash
    Remove              Kube.container_image
    Remove              Kube.container_name
    Remove              Kube.docker_id
    Remove              Kube.host
    Remove              Kube.pod_id

[FILTER]
    Name                nest
    Match               user-api.*
    Operation           nest
    Wildcard            Kube.*
    Nested_under        kubernetes
    Remove_prefix       Kube.

[OUTPUT]
    Name                cloudwatch_logs
    Match               user-api.*
    region              ${AWS_REGION}
    log_group_name      /aws/containerinsights/${CLUSTER_NAME}/user-api
    log_stream_prefix   app-
    auto_create_group   true
    extra_user_agent    container-insights