[英]Failed to execute user defined function in Spark-Scala
Below is the UDF to convert multivalued column into map.下面是将多值列转换为映射的 UDF。
def convertToMapFn (c: String): Map[String,String] = {
val str = Option(c).getOrElse(return Map[String, String]())
val arr = str.split(",")
val l = arr.toList
val regexPattern = ".*(=).*".r
s"$c".toString match {
case regexPattern(a) => l.map(x => x.split("=")).map(a => {if(a.size==2) (a(0).toString -> a(1).toString) else "ip_adr" -> a(0).toString} ).toMap
case "null" => Map[String, String]()
}
}
val convertToMapUDF = udf(convertToMapFn _)
I am able to display the data, but while trying to insert the data into Delta table, I am getting the below error.我能够显示数据,但是在尝试将数据插入 Delta 表时,出现以下错误。
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 97.0 failed 4 times, most recent failure: Lost task 9.3 in stage 97.0 (TID 2561, 10.73.244.39, executor 5): org.apache.spark.SparkException: Failed to execute user defined function($read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$Lambda$2326/1884779796: (string) => map<string,string>)
Caused by: scala.MatchError: a8:9f:e (of class java.lang.String)
at line396de0100d5344c9994f63f7de7884fe49.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.convertToMapFn
Caused by: org.apache.spark.SparkException: Failed to execute user defined function($read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$Lambda$2326/1884779796: (string) => map<string,string
Could someone pls let me know how to fix this.有人可以让我知道如何解决这个问题。 Thank you谢谢
You can see in the error message that you have a MatchError
.您可以在错误消息中看到您有一个MatchError
。 This happens when you don't account for all possible match cases.当您不考虑所有可能的匹配情况时,就会发生这种情况。 A basic fix is to change case "null" =>
to case _ =>
which will match anything that the regex doesn't.一个基本的修复是将case "null" =>
更改为case _ =>
这将匹配正则表达式不匹配的任何内容。
Other matters:其他事项:
s"$c".toString
is equivalent to writing c
in this case. s"$c".toString
在这种情况下相当于写c
。str
and not c
我认为你的意思是匹配str
而不是c
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.