从 Datadog 摄取中排除日志

Question

我有一个运行 datadog 和一些微服务的 kube.netes 集群。 每个微服务每 5 秒进行一次健康检查，以确保服务正常运行。 我想排除这些健康检查日志被摄入 Datadog。

我想我需要使用log_processing_rules并且我已经尝试过了，但是健康检查日志仍在进入 Datadog 的日志部分。 我当前的部署如下所示：

apiVersion: apps/v1
kind: Deployment
[ ... SNIP ... ]
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
        version: "fac8fb13"
      annotations:
        rollme: "IO2ad"
        tags.datadoghq.com/env: development
        tags.datadoghq.com/version: "fac8fb13"
        tags.datadoghq.com/service: my-service
        tags.datadoghq.com/my-service.logs: |
          [{
            "source": my-service,
            "service": my-service,
            "log_processing_rules": [
              {
                "type": "exclude_at_match",
                "name": "exclude_healthcheck_logs",
                "pattern": "\"RequestPath\": \"\/health\""
              }
            ]
          }]

以及来自 kube.netes pod 的日志：

$ kubectl logs my-service-pod

{
  "@t": "2022-01-07T19:13:05.3134483Z",
  "@m": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
  "@i": "REDACTED",
  "ElapsedMilliseconds": 7.5992,
  "StatusCode": 200,
  "ContentType": "text/plain",
  "ContentLength": null,
  "Protocol": "HTTP/1.1",
  "Method": "GET",
  "Scheme": "http",
  "Host": "10.64.0.80:5000",
  "PathBase": "",
  "Path": "/health",
  "QueryString": "",
  "HostingRequestFinishedLog": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
  "EventId": {
    "Id": 2,
    "Name": "RequestFinished"
  },
  "SourceContext": "Microsoft.AspNetCore.Hosting.Diagnostics",
  "RequestId": "REDACTED",
  "RequestPath": "/health",
  "ConnectionId": "REDACTED",
  "dd_service": "my-service",
  "dd_version": "54aae2b5",
  "dd_env": "development",
  "dd_trace_id": "REDACTED",
  "dd_span_id": "REDACTED"
}

编辑：删除了上面log_processing_rules数组的第二个元素，因为我已经尝试使用 rules 数组中的 1 和 2 元素。

EDIT2：我还尝试将log_processing_rules类型更改为 INCLUDE at match 以试图解决这个问题：

"log_processing_rules": [
  {
    "type": "include_at_match",
    "name": "testing_include_at_match",
    "pattern": "somepath"
  }
]

我仍然在 Datadog 中获取健康日志（理论上我不应该因为/health不是匹配模式的一部分）

Answer 1

我认为问题在于您要定义多种模式； 文档state， If you want to match one or more patterns you must define them in a single expression.

尝试这样的事情，看看会发生什么：

"log_processing_rules": [
  {
    "type": "exclude_at_match",
    "name": "exclude_healthcheck_logs",
    "pattern": "\/health|\"RequestPath\": \"\/health\""
  }

Answer 2

所有这些答案都是正确的，但我的具体问题是source和service的 Datadog 注释没有被正确引用：

        ad.datadoghq.com/my-service.logs: |
          [{
            "source": "my-service",    # Needs Quotes
            "service": "my-service",   # Needs Quotes
            "log_processing_rules": [
              {
                "type": "exclude_at_match",
                "name": "exclude_healthcheck_logs",
                "pattern": "\"RequestPath\": \"\/health\""
              }
            ]
          }]

Answer 3

确保您将 label com.datadoghq.ad.logs应用到生成日志消息的服务容器...而不是Datadog 代理容器。 （那是我的错误。）