在 ELK 中，如何防止 filebeat 将多个日志行分组/集群到单个文档中？

Question

我正在尝试将日志索引到 elasticsearch。

为了支持我的 Java 日志的多行，我在 filebeat.yml 中包含了以下配置。 我不确定 grok 模式是否适用于filebeat ，但我的要求是日志应仅以时间戳开头！

filebeat.prospectors:
  - type: log
    paths:
      - /ofbiz/runtime/logs/*.log
    fields:
      docType: app-logs
    document_type: app-logs
    multiline.type: pattern
    multiline.pattern: "^%{TIMESTAMP_ISO8601}"
    multiline.negate: true
    multiline.match: after
    ignore_older: 24h
    close_inactive: 60m

fields:
  env: dev
  project: Ofbiz-Backend

output:
  logstash:
    enabled: true
    hosts:
      - elk:5044
    timeout: 15
    ssl.enabled: false

但是我可能会遗漏一些东西，因此成批的预期日志行包含在单个文档中。

我的预期结果是这些日志应该单独索引！

自上周以来，我尝试了所有我探索过的解决方案，其中包括将流式日志直接传输到弹性搜索和使用摄取管道，将negate和match的配置更改为每个可能的组合！ 我没能成功！

示例日志行如下：

2021-11-26 08:35:04,719 |OFBiz-JobQueue-0     |PersistedServiceJob           |E| PersistedServiceJob.getContext(): SAXException
org.xml.sax.SAXParseException: Character reference "&#55357" is an invalid XML character.
    at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) ~[xercesImpl-2.9.1.jar:?]
    at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) ~[xercesImpl-2.9.1.jar:?]
    at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121) ~[?:1.8.0_275]
    at org.apache.ofbiz.base.util.UtilXml.readXmlDocument(UtilXml.java:457) ~[ofbiz.jar:?]
    at org.apache.ofbiz.base.util.UtilXml.readXmlDocument(UtilXml.java:371) ~[ofbiz.jar:?]
    at org.apache.ofbiz.entity.serialize.XmlSerializer.deserialize(XmlSerializer.java:97) ~[ofbiz.jar:?]
    at org.apache.ofbiz.service.job.PersistedServiceJob.getContext(PersistedServiceJob.java:293) [ofbiz.jar:?]
    at org.apache.ofbiz.service.job.GenericServiceJob.exec(GenericServiceJob.java:70) [ofbiz.jar:?]
    at org.apache.ofbiz.service.job.AbstractJob.run(AbstractJob.java:87) [ofbiz.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2021-11-26 08:35:04,719 |OFBiz-JobQueue-0     |PersistedServiceJob           |E| Job context is null
2021-11-26 08:35:04,719 |OFBiz-JobQueue-0     |ModelService                  |E| [ModelService.validate] : {sendProjectStatusNotification} : (IN) Required test error: org.apache.ofbiz.service.ServiceValidationException: The following required parameter is missing: [IN] [sendProjectStatusNotification.createdByPartyId]The following required parameter is missing: [IN] [sendProjectStatusNotification.projectId]The following required parameter is missing: [IN] [sendProjectStatusNotification.statusId]The following required parameter is missing: [IN] [sendProjectStatusNotification.title]The following required parameter is missing: [IN] [sendProjectStatusNotification.toRecipients]
2021-11-26 08:35:04,719 |OFBiz-JobQueue-0     |ServiceDispatcher             |E| Incoming context (in runSync : sendProjectStatusNotification) does not match expected requirements