流利的Openstack Log Regex格式

Question

我試圖解析來自所有OpenStack服務的日志，並將其發送到JSON中的S3。

我能夠使用這種多行格式來解析日志。

<source>
  @type tail
  path /var/log/nova/nova-api.log
  tag nova
  format multiline
  format_firstline /\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}(\.\d{3,6})? [^ ]* [^ ]* [^ ]*/
  format1 /(?<DateTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})(\.\d{3,6})? (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) (\[(?<context>[^\]]*)\])? (?<message>.*)/
  time_format %F %T.%L
</source>

解析這種類型的日志

2018-09-05 12:34:01.451 3212169 INFO nova.osapi_compute.wsgi.server [req-0f15c395-5962-4a14-ba7a-730f86d6eb7e f1ab53782de846798920050941f3bcff f5d0c8e495cd4793814f8b979693ac17 - default default] 192.168.1.1 "GET /v2.1/os-services HTTP/1.1" status: 200 len: 1435 time: 0.0372221

而這種類型

2018-09-05 12:34:33.631 2813186 INFO nova.api.openstack.placement.requestlog [req-d4573763-4521-419f-a4ba-c489a7e17ea9 fba548d81cd6480b90940a89e00f4133 6ba1bbedb40c411e9482128150886ba6 - default default] 192.168.1.1 "DELETE /allocations/196db945-0089-40ae-8108-a950fb453296" status: 204 len: 0 microversion: 1.0
AH01626: authorization result of Require all granted: granted
AH01626: authorization result of <RequireAny>: granted
AH01626: authorization result of Require all granted: granted
AH01626: authorization result of <RequireAny>: granted
2018-09-05 12:34:37.737 2813187 INFO nova.api.openstack.placement.requestlog [req-08b8b3e4-6981-4cd4-b90b-c922b7002b28 fba548d81cd6480b90940a89e00f4133 6ba1bbedb40c411e9482128150886ba6 - default default] 192.168.1.1 "GET /allocation_candidates?limit=1000&resources=CUSTOM_Z270_A%3A1" status: 200 len: 473 microversion: 1.17

進入這個

流利的演示

我正在嘗試使用此獲取ip，request_type路徑和狀態

(?<DateTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})(\.\d{3,6})? (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) (\[(?<context>[^\]]*)\]) (?<ip>[[\d+\.]*) \"(?<http_request_type>[^ \"]+) (?<http_request_path>[^\"]+)\" status\: (?<http_status_code>[^ ]+) (?<message>.*)

在這里查看演示。

Regex101演示

可以在regex101中正常運行，但是td-agent失敗並顯示此錯誤

[error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Invalid regexp in format1: premature end of char-class: /(?<DateTime>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2})(\\.\\d{3,6})? (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) (\\[(?<context>[^\\]]*)\\]) (?<ip>[[\\d+\\.]*) \\\"(?<http_request_type>[^ \\\"]+) (?<http_request_path>[^\\\"]+)\\\" status\\: (?<http_status_code>[^ ]+) (?<message>.*)/m

Answer 1

嘗試在線轉義xml，例如： https : //www.freeformatter.com/xml-escape.html或將@替換為&

Answer 2

問題是比賽組ip中有一個額外的方括號。 正確的模式是。

(?<DateTime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})(\.\d{3,6})? (?<pid>[^ ]*) (?<loglevel>[^ ]*) (?<class>[^ ]*) (\[(?<context>[^\]]*)\])? (?<ip>[\d+\.]*) \"(?<http_request_type>[^ \"]+) (?<http_request_path>[^\"]+)\" status\: (?<http_status_code>[^ ]+) (?<message>.*)