如何使用Spark讀取自定義多行日志

Question

我正在嘗試使用帶有spark的正則表達式模式解析自定義日志文件：

我的日志文件：

2018-04-11 06:27:36 localhost debug: localhost received discover from 0.0.0.0
2018-04-11 06:27:36 localhost debug:     sec = 0.4
2018-04-11 06:27:36 localhost debug:     Msg-Type = text
2018-04-11 06:27:36 localhost debug:     Content = XXXXXXXXXX
2018-04-11 06:27:34 localhost debug: localhost sending response to 0.0.0.0
2018-04-11 06:27:34 localhost debug:     sec = 0.3
2018-04-11 06:27:34 localhost debug:     Msg-Type = text
2018-04-11 06:27:34 localhost debug:     Content = XXXXXXXXXX
...

這是我的代碼片段：

case class Rlog(dateTime: String, server_name: String, log_type: String, server_addr:String, action: String, target_addr:String, cost:String, msg_type:String, content:String)
case class Slog(dateTime: String, server_name: String, log_type: String, server_addr:String, action: String, target_addr:String, msg_type:String, content:String)

val pattern_1 = """([\w|\s|\:|-]{19})\s([a-z]+)\s(\w+):\s(\w+)\sreceived\s(\w+)\sfrom\s([\.|\w]+)"""
val pattern_2 = """([\w|\s|\:|-]{19})\s([a-z]+)\s(\w+):\s{5}([\w|-]+)\s=\s([\.|\w]+)"""
val pattern_3 = """([\w|\s|\:|-]{19})\s([a-z]+)\s(\w+):\s(\w+)\ssending\s(\w+)\sto\s([\.|\w]+)"""

sc.textFile("/directory/logfile").map(?????)

有沒有辦法做到這一點？

Answer 1

您可以在map使用pattern.unapplySeq(string)來獲取與正則表達式匹配的所有組匹配的List 。

例如，如果您有字符串：

val str = "2018-04-11 06:27:36 localhost debug: localhost received discover from 0.0.0.0"

然后你運行：

pattern_1.unapplySeq(str)

你會得到：

 Option[List[String]] = Some(List(2018-04-11 06:27:36, localhost, debug, localhost, discover, 0.0.0.0))

我已經將您的示例用於此解決方案。 此答案假定某個日志類型以及與之關聯的msg類型，內容和秒數都將使用相同的時間戳打印。

// case class defintions here
// regex pattern_1, pattern_2, pattern_3 defined here

val rdd = sc.textFile("file").cache

// Filter in 3 rdds based on the pattern that gets matched
val receivedRdd = rdd.filter(_.matches(pattern_1.toString)).map(pattern_1.unapplySeq(_).get)
val sentRdd = rdd.filter(_.matches(pattern_3.toString)).map(pattern_3.unapplySeq(_).get)
val otherRdd = rdd.filter(_.matches(pattern_2.toString)).map(pattern_2.unapplySeq(_).get)

// Convert it to a dataframe
// Names are matching with case class Rlog and Slog
// To facilitate the conversion to Datasets

val receivedDF = receivedRdd.map{ case List(a,b,c,d,e,f) => (a,b,c,d,e,f)}
                            .toDF("dateTime" , "server_name", "log_type", "server_addr", "action", "target_addr")

val sentDF = sentRdd.map{ case List(a,b,c,d,e,f) => (a,b,c,d,e,f)}
                    .toDF("dateTime" , "server_name", "log_type", "server_addr", "action", "target_addr")

// Convert multiple lines containing msg-type, content etc to single line using pivot
val otherDF = otherRdd.map{ case List(ts , srvr, typ, i1 , i2) => (ts , srvr, typ, i1 , i2) }
                      .toDF("dateTime" , "server_name", "log_type", "i1" , "i2")
                      .groupBy("dateTime" , "server_name", "log_type")
                      .pivot("i1").agg(first($"i2") )
                      .select($"dateTime", $"server_name", $"log_type", $"sec".as("cost") , $"Msg-Type".as("msg_type"), $"Content".as("content"))

otherDF.show
//+-------------------+-----------+--------+----+--------+----------+
//|           dateTime|server_name|log_type|cost|msg_type|   content|
//+-------------------+-----------+--------+----+--------+----------+
//|2018-04-11 06:27:34|  localhost|   debug| 0.3|    text|XXXXXXXXXX|
//|2018-04-11 06:27:36|  localhost|   debug| 0.4|    text|XXXXXXXXXX|
//+-------------------+-----------+--------+----+--------+----------+

// Finally join based on dateTime, server_name and log_type and convert to Datasets

val RlogDS = receivedDF.join(otherDF, Seq("dateTime" , "server_name", "log_type")).as[Rlog]
val SlogDS = sentDF.join(otherDF, Seq("dateTime" , "server_name", "log_type")).as[Slog]

RlogDS.show(false)
//+-------------------+-----------+--------+-----------+--------+-----------+----+--------+----------+
//|           dateTime|server_name|log_type|server_addr|  action|target_addr|cost|msg_type|   content|
//+-------------------+-----------+--------+-----------+--------+-----------+----+--------+----------+
//|2018-04-11 06:27:36|  localhost|   debug|  localhost|discover|    0.0.0.0| 0.4|    text|XXXXXXXXXX|
//+-------------------+-----------+--------+-----------+--------+-----------+----+--------+----------+

SlogDS.show(false)
//+-------------------+-----------+--------+-----------+--------+-----------+----+--------+----------+
//|dateTime           |server_name|log_type|server_addr|action  |target_addr|cost|msg_type|content   |
//+-------------------+-----------+--------+-----------+--------+-----------+----+--------+----------+
//|2018-04-11 06:27:34|localhost  |debug   |localhost  |response|0.0.0.0    |0.3 |text    |XXXXXXXXXX|
//+-------------------+-----------+--------+-----------+--------+-----------+----+--------+----------