為自定義時間格式創建 Logstash 輸入和 grok 過濾器

Question

已經快兩個月了，我無法弄清楚如何使以下日志進行解析。 面臨的挑戰：

1. 日志周圍有雙引號，並且日志的格式不是很一致
2. 日志之間有許多制表符和奇數空格

感謝任何關於如何開始的指南

"[5/10/22 16:07:39:393 GTS] 00000330 SystemErr     R    at com.ibm.mdr.DrStateMgr.eventFromUser(DrStateMgr.java:2952)"
"[5/10/22 16:07:39:393 GTS] 00000330 SystemErr     R    at com.ibm.mdr.DrStateMgr.dequeueAndFireEvents(DrStateMgr.java:5010)"
[5/10/22 16:03:49:982 GTS] 000000a4 WebContainer  E com.ibm.ws.webcontainer.internal.WebContainer handleRequest TEST_SERVER: A WebGroup/Virtual Host to handle / has not been defined.
[5/8/22 6:43:42:236 GTS] 00000001 SSLConfigMana W   AAPKI0003A: The runtime has at least one SSL configuration that supports only weak TLSv1 or TLSv1.1 handshake protocols. For increased security, modify the configuration to use only stronger protocols such as TLSv1.2 or later. Find instructions to update your configuration at https://www.ibm.com/support/pages/node/1077951. SSL configurations that use the weaker SSL protocols include: [XDADefaultSSLSettings((cell):AFDJP01PCell01)].
[5/8/22 6:43:42:220 GTS] 00000001 WSKeyStore    W   SSPKI0002A: One or more key stores are using the default password.
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I   DDPKI0004A: The process has the java security property jdk.tls.disabledAlgorithms set to [SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, DH keySize < 1024, DESede, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, DES_CBC].  The WebSphere Application server is setting the java security property jdk.tls.disabledAlgorithms to [SSLv3, RC4, DH keySize < 768, MD5withRSA]. 
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I   DDPKI0004A: The process has the java security property jdk.certpath.disabledAlgorithms set to [MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224].  The WebSphere Application server is setting the java security property jdk.certpath.disabledAlgorithms to [MD2, RSA keySize < 1024, MD5]. 
[5/8/22 6:43:42:204 GTS] 00000001 FIPSManager   I   EEPKI0005A: FIPS security mode is : No FIPS property found. 
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I   GGPKI0007A: The SSL configuration is initializing.
[5/8/22 6:43:42:189 GTS] 00000001 SSLComponentI I   HHPKI0008A: SSL service is initializing the configuration
[5/8/22 6:43:42:095 GTS] 00000001 PluginConfigS I   PLGC0044B: The plug-in configuration service started successfully.
[5/8/22 6:43:41:345 GTS] 00000001 AdminInitiali A   ADMN0054E: The administration service is initialized.
[5/8/22 6:43:41:048 GTS] 00000001 ProviderTrack I com.ibm.ffdc.osgi.ProviderTracker AddingService FFDC1007I: FFDC Provider Installed: com.ibm.ws.ffdc.impl.FfdcProvider@ed46329b
[5/8/22 6:43:40:908 GTS] 00000001 ComponentMeta I   ASVR0150U: The runtime provisioning feature is disabled. All components will be started.
[5/8/22 6:43:39:923 GTS] 00000001 ModelMgr      I   ASVR0180U: Initializing core configuration models
[5/8/22 6:43:39:783 GTS] 00000001 ManagerAdmin  I   TRAS0555T: The message IDs that are in use are deprecated
[5/8/22 6:43:39:783 GTS] 00000001 ManagerAdmin  I   TRAS0787K: The startup trace state is *=info.
"[5/8/22 7:37:18:809 GTS]     FFDC Exception:java.io.FileNotFoundException SourceId:com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -IOE ProbeId:1044
java.io.FileNotFoundException: DEAV0180D: File not found: /favicon.ico
    at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
    at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:958)
    at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:486)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
    at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4075)
    at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1019)
    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:213)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Reporter:null"

預期 output

{
  "month": [
    [
      "5"
    ]
  ],
  "day": [
    [
      "10"
    ]
  ],
  "year": [
    [
      "22"
    ]
  ],
  "time": [
    [
      "16:03:49:982"
    ]
  ],
  "instance": [
    [
      "000000a4"
    ]
  ]
    "process": [
    [
      "WebContainer  E com.ibm.ws.webcontainer.internal.WebContainer handleRequest TEST_SERVER: A WebGroup/Virtual Host to handle / has not been"
    ]
  ]
    "server": [
    [
      "TEST_SERVER"
    ]
  ]
    "error": [
    [
      "A WebGroup/Virtual Host to handle / has not been"
    ]
  ]
}

使用中的 Grok 模式

\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{GREEDYDATA:host} 

Answer 1

對於單個日志中的這種多重模式，您可以嘗試如下所示。 如果任何 grok 解析失敗，可能是由於中間有額外的空間，因此構造它並在其中添加這些模式。

    filter
    {
    grok
    {
   match => {"message" => ["\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{DATA:host} %{WORD:source}  %{WORD:logtype} %{DATA:lib} %{WORD:request_type} %{DATA:server}: %{GREEDYDATA:detailed_message}","\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{DATA:host} %{WORD:source} %{WORD:logtype} %{DATA:code}: %{GREEDYDATA:detailed_message}"]}
    }
    }

但是，上述模式僅適用於 W,R,I,A 日志類型的單行，但不適用於多行

IE，

 "[5/8/22 7:37:18:809 GTS]     FFDC Exception:java.io.FileNotFoundException SourceId:com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -IOE ProbeId:1044
java.io.FileNotFoundException: DEAV0180D: File not found: /favicon.ico
    at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
    at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:958)
    at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:486)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
    at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4075)
    at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1019)
    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:213)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
    at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
    at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Reporter:null"

保持發布您的反饋！ 謝謝 ！！！