[英]Spark Dataframe UDF - Schema for type Any is not supported
我正在編寫 Spark Scala UDF 並面臨“java.lang.UnsupportedOperationException:不支持任何類型的架構”
import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.functions.udf
val aBP = udf((bG: String, pS: String, bP: String, iOne: String, iTwo: String) => {
if (bG != "I") {"NA"}
else if (pS == "D")
{if (iTwo != null) iOne else "NA"}
else if (pS == "U")
{if (bP != null) bP else "NA"}
})
這是拋出錯誤“java.lang.UnsupportedOperationException:不支持任何類型的架構”
正如在此鏈接中討論的那樣,您的 udf 應該返回:
因此,如果您在代碼中添加另一個 else,編譯將成功。
val aBP = udf((bG: String, pS: String, bP: String, iOne: String, iTwo: String) => {
if (bG != "I") {"NA"}
else if (pS == "D") {
if (iTwo != null)
iOne
else "NA"
} else if (pS == "U") {
if (bP != null)
bP
else
"NA"
} else {
""
}
})
您還可以使用模式匹配重新分發您的代碼:
val aBP = udf [String, String, String, String, String, String] {
case (bG: String, _, _, _, _) if bG != "I" => "NA"
case (_, pS: String, _, iOne: String, iTwo: String) if pS == "D" && iTwo.isEmpty => iOne
case (_, pS: String, _, _, _) if pS == "D" => "NA"
case (_, pS: String, bP: String, _, _) if pS == "U" && bP.isEmpty => bP
case (_, pS: String, _, _, _) if pS == "U" => "NA"
case _ => ""
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.