Benthos grok 日志解析

Question

所以我有这个日志，我试图用 benthos grok 解析它。 我需要做的是返回 json 5 个元素： • 时间戳 • 连接方向（入站/出站） • 源 IP • 目标 IP • json 此日志格式的源端口：

<134>1 2023-01-21T17:18:05Z CHKPGWMGMT CheckPoint 16575 - [action:"Accept"; flags:"411908"; ifdir:"outbound"; ifname:"eth0"; logid:"0"; loguid:"{0x80c5f24,0x273f572f,0x1a6c6aae,0x5f835b6e}"; origin:"10.21.10.2"; originsicname:"cn=cp_mgmt,o=CHKPGWMGMT..f6b99b"; sequencenum:"4"; time:"1674314285"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={F7CAC520-C428-484E-8004-06A1FAC151A3};mgmt=CHKPGWMGMT;date=1667399823;policy_name=Standard]"; dst:"10.21.10.2"; inzone:"Local"; layer_name:"Network"; layer_uuid:"8a994dd3-993e-4c0c-92a1-a8630b153f4c"; match_id:"1"; parent_rule:"0"; rule_action:"Accept"; rule_uid:"102f52bf-da21-49cd-b2e2-6affe347215d"; outzone:"Local"; product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"46540"; service:"1433"; service_id:"https"; src:"10.21.9.1"]

input:
  type: file
  file:
    paths: [./intput.txt]
    codec: lines

pipeline:
  processors:
    - grok:
        expressions:
          - '%{NGFWLOGFILE}'
        pattern_definitions:
            NGFWLOGFILE: '%{NOTSPACE:interfaceid} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:Letters} %{NOTSPACE:Mhm} %{NOTSPACE:Skaicius} %{NOTSPACE:AA} %{NOTSPACE:Action}'
#    - mapping: |
#       root.timestamp = this.timestamp
#        root.Action = this.Action

output:
  stdout: {}
#output:
 # label: ""
 # file:
 #   path: "Output.txt"
 #  codec: lines

所以我尝试使用grok将日志解析为json格式并映射以过滤我想要获取的部分。 我被卡住的部分是 pattern_definitions 如何从日志文件中已有名称的列表中提取数据，或者我应该使用一些更好的方法来完成任务？

Answer 1

Grok 在幕后转换为正则表达式，所以我认为它没有任何列表等概念。 尝试这个：

input:
  generate:
    count: 1
    interval: 0s
    mapping: |
      root = """<134>1 2023-01-21T17:18:05Z CHKPGWMGMT CheckPoint 16575 - [action:"Accept"; flags:"411908"; ifdir:"outbound"; ifname:"eth0"; logid:"0"; loguid:"{0x80c5f24,0x273f572f,0x1a6c6aae,0x5f835b6e}"; origin:"10.21.10.2"; originsicname:"cn=cp_mgmt,o=CHKPGWMGMT..f6b99b"; sequencenum:"4"; time:"1674314285"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={F7CAC520-C428-484E-8004-06A1FAC151A3};mgmt=CHKPGWMGMT;date=1667399823;policy_name=Standard]"; dst:"10.21.10.2"; inzone:"Local"; layer_name:"Network"; layer_uuid:"8a994dd3-993e-4c0c-92a1-a8630b153f4c"; match_id:"1"; parent_rule:"0"; rule_action:"Accept"; rule_uid:"102f52bf-da21-49cd-b2e2-6affe347215d"; outzone:"Local"; product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"46540"; service:"1433"; service_id:"https"; src:"10.21.9.1"]"""

pipeline:
  processors:
    - grok:
        expressions:
          - "%{NGFWLOGFILE}"
        pattern_definitions:
          NGFWLOGFILE: |-
            %{NOTSPACE:interfaceid} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:Letters} %{NOTSPACE:Mhm} %{NOTSPACE:Skaicius} %{NOTSPACE:AA} \[%{GREEDYDATA}; ifdir:"%{DATA:connectionDirection}"; %{GREEDYDATA}; dst:"%{DATA:destinationIP}"; %{GREEDYDATA}; s_port:"%{DATA:sourcePort}"; %{GREEDYDATA}; src:"%{DATA:sourceIP}"\]

output:
  stdout: {}