堆栈内存溢出
  繁体   English   中英

更正输入日志的正则表达式

[英]Correct regular expression for the input log

 原文  2017-08-24 12:42:14  3   1    regex/ logstash/ multiline/ logstash-grok/ logstash-configuration

问题描述

输入日志看起来像这样，其中包含的数据是“|” sperated。 数据包含id | 类型| 要求| 响应 

110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
210000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
340000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
450000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
590000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

期望的输出：

第1日志： 

id- 110000

type-read

request-<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>

response-<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

第二次登录： 

id - 210000

type - read

request -

response -

同样对于“n”没有。 的日志

使用的配置文件： 

input {
  file {
    path => "/opt/test5/practice_new/final_xml.dat"
    start_position => "beginning"
    codec => multiline {
            pattern => "^%{NUMBER:method_id}\|%{DATA:method_type}\|<soapenv:Envelope>"
            negate => true
            what => previous
        }
  }
}
filter {
  grok {
    match => [ "message", "(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request><soapenv:Envelope>.*?</soapenv:Envelope>)\|(?<response><soapenv:Envelope>.*?</soapenv:Envelope>)" ]
  }
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "final"
  }
stdout {}
}

我尝试在Grok中使用正则表达式，但当前的表达式不适用于输入日志。

请帮我正则表达式。

1 个解决方案

解决方案1
 3  2017-08-24 12:46:01

你当前使用的正则表达式是(?m)^(?<method_id>\\d+)\\|(?<method_type>\\w+)\\|(?<request><soapenv:Envelope>.*?</soapenv:Envelope>)\\|(?<response><soapenv:Envelope>.*?</soapenv:Envelope>) ，它只能解析第3和第4列，如果它们以<soapenv:Envelope>开头并以</soapenv:Envelope>结束</soapenv:Envelope>有| 之间。

看来你需要一个正则表达式，将第3列标识为除|之外的任何字符序列 并且第4列应该除了|之外的任何数量的字符 直到新行后跟一个或多个数字然后| 。

采用 

(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request>[^|]*)\|(?<response>[^|\n]*(?:\n(?!\d+\|)[^|\n]*)*)

请参阅正则表达式演示 。

细节

  • (?m) - 制作的Ruby修饰符. 匹配线休息字符
  • ^ - 开始一行
  • (?<method_id>\\d+) - 组“方法”：一个或多个数字
  • \\| - 管道炭
  • (?<method_type>\\w+) - 组“method_type”：一个或多个字母，数字或_
  • \\| - 管道
  • (?<request>[^|]*) - 组“请求”：除|以外的任何0+字符
  • \\| - 管道
  • (?<response>[^|\\n]*(?:\\n(?!\\d+\\|)[^|\\n]*)*) - 组“响应”：
    • [^|\\n]* - 除|以外的任何0+字符 和LF（换行）
    • (?:\\n(?!\\d+\\|)[^|\\n]*)* - 0+次出现：
      • \\n - 换行符
      • (?!\\d+\\|) - 未跟随1+位+ |
      • [^|\\n]* - 除|以外的任何0+字符 和LF（换行）

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 什么是正确的正则表达式 正确的正则表达式 这个Java正则表达式正确吗？ 这种正则表达式的简化是否正确？ 日志文件的正则表达式 使用正则表达式解析日志 日志的正则表达式 空白输入是否有正则表达式？ 正则表达式的输入掩码 正则表达式掩码输入
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM