![](/img/trans.png)
[英]Scala UDF fails when called from a SELECT statement in DataBricks / Spark
[英]spark udf not being called
给定以下示例:
import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.functions._
val testUdf: UserDefinedFunction = udf((a: String, b: String, c: Int) => {
val out = s"test1: $a $b $c"
println(out)
out
})
val testUdf2: UserDefinedFunction = udf((a: String, b: String, c: String) => {
val out = s"test2: $a $b $c"
println(out)
out
})
Seq(("hello", "world", null))
.toDF("a", "b", "c")
.withColumn("c", $"c" cast "Int")
.withColumn("test1", testUdf($"a", $"b", $"c"))
.withColumn("test2", testUdf2($"a", $"b", $"c"))
.show
testUdf
似乎没有被调用。 没有错误,没有警告,它只是返回 null。
有没有办法检测这些静默故障? 另外,这里发生了什么?
火花 2.4.4 Scala 2.11
Scala 类型“Int”不允许空值。 变量“c”类型可以更改为“Integer”。
我不知道这是什么原因造成的。 但我认为这很可能是因为隐式转换
代码1
val spark = SparkSession.builder()
.master("local")
.appName("test")
.getOrCreate()
import spark.implicits._
val testUdf: UserDefinedFunction = udf((a: String, b: String, c: Int) => {
val out = s"test1: $a $b $c"
println(out)
out
})
Seq(("hello", "world", null))
.toDF("a", "b", "c")
.withColumn("test1", testUdf($"a", $"b", $"c"))
.show
代码2
val spark = SparkSession.builder()
.master("local")
.appName("test")
.getOrCreate()
import spark.implicits._
val testUdf: UserDefinedFunction = udf((a: String, b: String, c: String) => {
val out = s"test1: $a $b $c"
println(out)
out
})
Seq(("hello", "world", null))
.toDF("a", "b", "c")
.withColumn("test1", testUdf($"a", $"b", $"c"))
.show
code1 逻辑计划
code2 逻辑计划
You should have a scala.MatchError: scala.Null
error when you try to cast to null, besides your definition of UDF doesn't work for me as I got a java.lang.UnsupportedOperationException: Schema for type AnyRef is not supported
when I尝试注册它。
那这个呢:
import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.functions._
def testUdf(a: String, b: String, c: Integer): String = {
val out = s"test1: $a $b $c"
println(out)
out
}
def testUdf2(a: String, b: String, c: String): String = {
val out = s"test2: $a $b $c"
println(out)
out
}
val yourTestUDF = udf(testUdf _)
val yourTestUDF2 = udf(testUdf2 _)
// spark.udf.register("yourTestUDF", yourTestUDF) // just in case you need it in SQL
spark.createDataFrame(Seq(("hello", "world", null.asInstanceOf[Integer])))
.toDF("a", "b", "c")
.withColumn("test1", yourTestUDF($"a", $"b", $"c"))
.withColumn("test2", yourTestUDF2($"a", $"b", $"c"))
.show(false)
Output:
test1: hello world null
test2: hello world null
+-----+-----+----+-----------------------+-----------------------+
|a |b |c |test1 |test2 |
+-----+-----+----+-----------------------+-----------------------+
|hello|world|null|test1: hello world null|test2: hello world null|
+-----+-----+----+-----------------------+-----------------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.