[英]Spark Scala UDF : java.lang.UnsupportedOperationException: Schema for type Any is not supported
I am trying to return a map from the UDF with in if else and getting the below exception, Any pointers please?我正在尝试从 UDF 返回一个 map 并在 if else 中得到以下异常,请指点?
java.lang.UnsupportedOperationException: Schema for type Any is not supported java.lang.UnsupportedOperationException:不支持任何类型的架构
import org.apache.spark.sql.functions.{col, udf}
import org.apache.spark.sql.functions._
val df2 = Seq(
("1", Map("Fld1" -> "USA","Fld2" -> "UK")),
("2", Map("Fld1" -> "Germany", "Fld2" -> "Portugal")),
("3", Map("Fld1" -> "France", "Fld2" -> "Paris"))
).toDF("id", "map")
val getmapUdf = udf((map1: Map[String, String]) => {
val fl1 = map1.getOrElse("Fld1","unknown")
val fl2 = map1.getOrElse("Fld2","unknown")
if (fl1 =="Germany")
{
Map("key1" -> "G")
}
else if(fl1 =="France")
{
if (fl2 =="UK")
{
Map("key1" ->"U")
}
else
{
Map("key1" ->"Y")
}
}
else if(fl1 =="France")
{
Map("key1" ->"G")
}
})
var temp2 = df2.withColumn("mymap", getmapUdf($"map"))
temp2.show(false)
You get that error because your UDF function doesn't always return a type Map[String,String]
, you are using if/else statements that do not cover default values when conditions are not satisfied, so the return type is Any.您收到该错误是因为您的 UDF function 并不总是返回类型Map[String,String]
,您使用的 if/else 语句在不满足条件时不涵盖默认值,因此返回类型为 Any。
However, you can do the same thing w/o UDF actually, using when
function:但是,您实际上可以在不使用 UDF 的情况下执行相同的操作,使用 function when
:
var temp2 = df2.withColumn(
"mymap",
when($"map" ("Fld1") === "Germany", map(lit("key1"), lit("G")))
when ($"map" ("Fld1") === "France" && $"map" ("Fld2") === "UK", map(lit("key1"),lit("G")))
when ($"map" ("Fld1") === "France", map(lit("key1"), lit("Y")))
)
temp2.show(false)
//+---+-----------------------------------+-----------+
//|id |map |mymap |
//+---+-----------------------------------+-----------+
//|1 |[Fld1 -> USA, Fld2 -> UK] |null |
//|2 |[Fld1 -> Germany, Fld2 -> Portugal]|[key1 -> G]|
//|3 |[Fld1 -> France, Fld2 -> Paris] |[key1 -> Y]|
//+---+-----------------------------------+-----------+
Anyway if you want to use UDF, modify the function to return Option[Map[String,String]]
.无论如何,如果您想使用 UDF,请修改 function 以返回Option[Map[String,String]]
。 Something like this:像这样的东西:
val getmapUdf = udf((map1: Map[String, String]) => {
val fl1 = map1.getOrElse("Fld1", "unknown")
val fl2 = map1.getOrElse("Fld2", "unknown")
if (fl1 == "Germany") {
Some(Map("key1" -> "G"))
} else if (fl1 == "France") {
if (fl2 == "UK") {
Some(Map("key1" -> "U"))
} else {
Some(Map("key1" -> "Y"))
}
} else if (fl1 == "France") {
Some(Map("key1" -> "G"))
} else {
None
}
})
I will just try to provide an alternative answer here if you want to keep using what you are doing but @blackbishop answer cover most of the options covered to implement the same stuff.如果您想继续使用您正在做的事情,我将尝试在这里提供一个替代答案,但@blackbishop 答案涵盖了实现相同内容的大部分选项。
In order for your code to work you can just make a change in your UDF to make sure you have else condition that returns a default map as shown below and then you won't get that error.为了使您的代码正常工作,您只需在 UDF 中进行更改,以确保您有返回默认 map 的其他条件,如下所示,然后您将不会收到该错误。
val getmapUdf = udf((map1: Map[String, String]) => {
val fl1 = map1.getOrElse("Fld1","unknown")
val fl2 = map1.getOrElse("Fld2","unknown")
if (fl1 =="Germany")
{
Map("key1" -> "G")
}
else if(fl1 =="France")
{
if (fl2 =="UK")
{
Map("key1" ->"U")
}
else
{
Map("key1" ->"Y")
}
}
else if(fl1 =="France")
{
Map("key1" ->"G")
}
else
{
Map("key1" -> "unknown")
}
})
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.