我正在尝试通过 Logstash 解析来自腾讯云的 CDN 日志文件。 没有我能找到的记录的 grok 格式

Question

日志行如下：

20221205110142 112.79.237.32 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=416044&end=617579&type=mpegts&resolution=640x360 202239 73 -1 200 NULL 5 "myapp/7.9.108 (Linux;Android 10) ExoPlayerLib/2.14.0" "416044-617579" GET HTTP/1.1 hit -
20221205110142 112.79.237.32 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=617580&end=826259&type=mpegts&resolution=640x360 209383 73 -1 200 NULL 4 "myapp/7.9.108 (Linux;Android 10) ExoPlayerLib/2.14.0" "617580-826259" GET HTTP/1.1 hit 0
20221205110143 112.79.237.32 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=826260&end=1029111&type=mpegts&resolution=640x360 203555 73 -1 200 NULL 4 "myapp/7.9.108 (Linux;Android 10) ExoPlayerLib/2.14.0" "826260-1029111" GET HTTP/1.1 hit 0
20221205110146 112.79.237.32 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=1029112&end=1235911&type=mpegts&resolution=640x360 207503 73 -1 200 NULL 6 "myapp/7.9.108 (Linux;Android 10) ExoPlayerLib/2.14.0" "1029112-1235911" GET HTTP/1.1 hit 0
20221205110150 112.79.237.32 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=1235912&end=1444027&type=mpegts&resolution=640x360 208819 73 -1 200 NULL 5 "myapp/7.9.108 (Linux;Android 10) ExoPlayerLib/2.14.0" "1235912-1444027" GET HTTP/1.1 hit 0
20221205110014 202.51.80.161 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=0&end=208491&type=mpegts&resolution=640x360 209195 35 -1 200 NULL 63 "myapp/7.9.108 (Linux;Android 11) ExoPlayerLib/2.14.0" "0-208491" GET HTTP/1.1 hit 0
20221205110015 202.51.80.161 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=208492&end=416043&type=mpegts&resolution=640x360 208255 35 -1 200 NULL 6 "myapp/7.9.108 (Linux;Android 11) ExoPlayerLib/2.14.0" "208492-416043" GET HTTP/1.1 hit 0
20221205110037 157.42.219.198 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_99338845_1.ts?start=826260&end=1029111&type=mpegts&resolution=640x360 203555 73 -1 200 NULL 5 "myapp/7.9.91 (Linux;Android 12) ExoPlayerLib/2.14.0" "826260-1029111" GET HTTP/1.1 hit 0
20221205110037 114.31.131.175 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_1319432614_5.ts?start=412848&end=623031&type=mpegts&resolution=640x360 210887 73 -1 200 NULL 5 "myapp/7.9.108 (Linux;Android 12) ExoPlayerLib/2.14.0" "412848-623031" GET HTTP/1.1 hit 0
20221205110041 114.31.131.175 xxxxxxxx.vod2.myqcloud.com /8d43a35evodsgpxxxxxxxx/zzzzzzzzzzzzzzzzzzzzzzzzz/624668012_1319432614_5.ts?start=623032&end=822311&type=mpegts&resolution=640x360 199983 73 -1 200 NULL 5 "myapp/7.9.108 (Linux;Android 12) ExoPlayerLib/2.14.0" "623032-822311" GET HTTP/1.1 hit 0

这不是其他标准 AFAIK 中定义的格式。

Answer 1

解析这些行的正则表达式是：

(?P<datetime>\d+) (?P<client_ip>[0-9\.]+) (?P<host>1301938931\.vod2\.myqcloud\.com) (?P<asset>[a-zA-Z0-9\.\/\_\?\=\&]+) (?P<bytes>\d+) (?P<district_code>\d+) (?P<isp_code>-?\d+) (?P<http_status>\d+) (?P<referer>\w+) (?P<request_time>\d+) (?P<ua>\".+?[\/\s][\d.]+\") (?P<byte_rabge>\"[\d\-]+\") (?P<method>\w+) (?P<protocol>[\w\/\.]+) (?P<miss>\w+) (?P<port>[\d\-]+)

对应的grok格式为：

%{DATESTAMP_EVENTLOG:log_timestamp} %{IPORHOST:clienthost} %{IPORHOST:site} %{URIPATH:path}(%{URIPARAM:params})? %{NUMBER:bytessent} %{NUMBER:district_code} %{NUMBER:isp_code} %{NUMBER:scstatus} %{IPORHOST:referrer} %{NUMBER:timetaken:int} %{QUOTEDSTRING:useragent} %{QUOTEDSTRING:byte_range} %{WORD:http_method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})? %{WORD:cache_status} %{WORD:port}