使用 logstash 解析多行 json 日志

Question

我需要使用 logstash 解析帶有異常和堆棧跟蹤的多行 json 日志。 我有使用 NLog 以 json 格式生成的 asp.net 核心日志：

{ "timestamp": "2021-01-31T17:18:30.1781670+03:00", "level": "Error", "eventid": "0", "logger": "WebScraper.WebApi.Controllers.ProductWatcherController", "callsite": "ProductWatcherController.CreateProduct", "message": "Internal server error", "exception": "System.Net.Http.HttpRequestException: The connection is not established, because the destination computer rejected the connection request.
 ---> System.Net.Sockets.SocketException (10061): The connection is not established. the destination computer rejected the connection request.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at WebScraper.Core.Helpers.HangfireSchedulerClient.CreateOrUpdateScheduler(ProductSchedulerDto productSchedulerDto) in D:\C#\WebScraper\WebScraper.Core\Helpers\HangfireSchedulerClient.cs:line 43
   at WebScraper.Core.ProductWatcherManager.CreateProduct(String productUrl, Site siteDto, List`1 scheduler, Boolean pushToHangfire) in D:\C#\WebScraper\WebScraper.Core\ProductWatcherManager.cs:line 131
   at WebScraper.WebApi.Controllers.ProductWatcherController.CreateProduct(CreateProductDto createProductDto) in D:\C#\WebScraper\WebScraper.WebApi\Controllers\ProductWatcherController.cs:line 202", "url": "https:\/\/localhost\/api\/ProductWatcher\/product", "action": "CreateProduct" }

我目前正在測試這個 logstash.conf

input {
        file {
                codec => multiline
                {
                    pattern => '^{'
                    negate => true
                    what => previous                
                }
                start_position => "beginning"
                path => "/usr/data/json-log.txt"
                sincedb_path => "/dev/null"
        }
}

filter {
        mutate {
            gsub => [ 'message','\n','']
        }
        json {
                source => "message"
        }
}

output {
        elasticsearch {
                hosts => "elasticsearch:9200"
                index => "file-json"
        }

        stdout {}
}

請幫助進行正確的配置。

Answer 1

JSON 解析器反對“D:\C#\WebScraper\WebScraper.Core”，因為 \C 不是有效的轉義。 在您的 logstash 日志中，您應該看到

:exception=>#<LogStash::Json::ParserError: Unrecognized character escape 'C' (code 67)

我建議您修改消息以用正斜杠替換反斜杠。 你可以使用

mutate { gsub => [ 'message', '[\\]', '/'] }

請注意，引號前不能有反斜杠，因為它被視為轉義，因此您必須使用包含反斜杠的字符組。 這也意味着您不能輕松地將\轉換為\\ ，這是修復它的另一種方法。