Elasticsearch：Grok 管道不工作（不適用於日志）

Question

我創建了一個管道：“ExtractOwaspErrorFields”，它帶有一個從消息字段中提取字段的 grok 處理器。 在 Kibana -> Stack Management -> Ingest Pipelines -> Test Pipeline 中對其進行測試時，它運行良好。 我通過提供文檔的 _id 和 _index 來使用真實的日志文檔。 處理器按預期從消息字段中提取字段。 當我將處理器添加到我的 filebeat.yml 時，“發現”中根本沒有顯示任何文檔：

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: xxx
  password: xxx
  pipeline: ExtractOwaspErrorFields

我的管道配置如下：

{
  "ExtractOwaspErrorFields" : {
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            "%{OWASP_ERRORLOG}"
          ],
          "pattern_definitions" : {
            "OWASP_ERRORLOG" : "\\[%{HTTPDERROR_DATE:timestamp}\\] \\[:%{LOGLEVEL:loglevel}\\] \\[pid %{POSINT:pid}:tid %{NUMBER:tid}\\]?( \\[client %{IPORHOST:client}:%{POSINT:clientport}\\]) \\[client %{IPORHOST:client_ip}\\] ModSecurity: ?(%{APACHE_ERROR_MESSAGE:error}) \\[file \"%{PATH:matching_rule_file}\"\\] \\[line \"%{POSINT:matching_rule_line}\"\\] \\[id \"%{NUMBER:matching_rule_id}\"\\] \\[msg \"%{DATA:owasp_message_string}\"\\] \\[data \"%{DATA:owasp_message_data}\"\\] \\[severity \"%{WORD:owasp_severity}\"\\] \\[ver \"%{DATA:owasp_version}\"\\] %{GREEDYDATA:tags} \\[hostname \"%{HOSTNAME:hostname}\"\\] \\[uri \"%{URIPATHPARAM:uri}\"\\] \\[unique_id \"%{DATA:unique_id}\"\\]",
            "APACHE_ERROR_MESSAGE" : "( .+?(?= \\[%{WORD} \"))"
          },
          "if" : "ctx?.docker.container.labels.com_docker_stack_namespace == 'modsecurity'",
          "ignore_failure" : true
        }
      }
    ]
  }
}

我用於測試管道的示例文檔是：

{
  "docs":
  [
    {
      "_id": "OzAKyoIBILrgz4V8VcpG",
      "_index": "filebeat-7.17.5-2022.08.02-000001",
      "_source": {
        "docker": {
          "container": {
            "labels": {
              "com_docker_stack_namespace": "modsecurity"
            }
          }
        },
        "ecs": {
          "version": "1.12.0"
        },
        "stream": "stderr",
        "message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \"(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\" at ARGS:test. [file \"/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"71\"] [id \"930110\"] [msg \"Path Traversal Attack (/../)\"] [data \"Matched Data: ../ found within ARGS:test: ../\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.3.2\"] [tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [hostname \"modsecurity\"] [uri \"/\"] [unique_id \"YwSeR-gNfcPQLAl5gStNfAAAAQE\"]"
      }
    }
  ]
}

管道測試結果：

{
  "docs": [
    {
      "doc": {
        "_index": "filebeat-7.17.5-2022.08.02-000001",
        "_type": "_doc",
        "_id": "OzAKyoIBILrgz4V8VcpG",
        "_source": {
          "owasp_severity": "CRITICAL",
          "owasp_message_string": "Path Traversal Attack (/../)",
          "pid": "226",
          "error": " Warning. Pattern match \"(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\" at ARGS:test.",
          "tid": "139758264993536",
          "clientport": "41062",
          "docker": {
            "container": {
              "labels": {
                "com_docker_stack_namespace": "modsecurity"
              }
            }
          },
          "owasp_version": "OWASP_CRS/3.3.2",
          "hostname": "modsecurity",
          "ecs": {
            "version": "1.12.0"
          },
          "stream": "stderr",
          "client": "10.0.1.48",
          "client_ip": "10.0.1.48",
          "timestamp": "Tue Aug 23 11:30:47.675452 2022",
          "unique_id": "YwSeR-gNfcPQLAl5gStNfAAAAQE",
          "matching_rule_id": "930110",
          "message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \"(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\" at ARGS:test. [file \"/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"71\"] [id \"930110\"] [msg \"Path Traversal Attack (/../)\"] [data \"Matched Data: ../ found within ARGS:test: ../\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.3.2\"] [tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [hostname \"waf\"] [uri \"/\"] [unique_id \"YwSeR-gNfcPQLAl5gStNfAAAAQE\"]",
          "uri": "/",
          "owasp_message_data": "Matched Data: ../ found within ARGS:test: ../",
          "tags": "[tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"]",
          "loglevel": "error",
          "matching_rule_file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf",
          "matching_rule_line": "71"
        },
        "_ingest": {
          "timestamp": "2022-08-23T12:08:05.5017157Z"
        }
      }
    }
  ]
}

有人可以告訴我我的管道有什么問題或我錯過了什么。 為什么它在“測試管道”中工作，但當我將它們用於 filebeat.yml 中的真實日志時卻沒有，非常感謝

Answer 1

通過將管道重命名為僅包含小寫字符來解決它。 顯然filebeat.yml的解析將管道值轉換為小寫，因此在彈性端找不到。