[英]Spark dataframe udf No TypeTag available
I am trying to extend spark
ml
pipeline model with a filter transformer, after我正在尝试使用过滤器变压器扩展spark
ml
管道模型,之后
abstract class RuleFilter[IN, T <: RuleFilter[IN, T]]
extends RuleTransformer with HasInputCol {
// def filterFuntion: String
/** @group setParam */
def setInputCol(value: String): T = set(inputCol, value).asInstanceOf[T]
protected def createFilterFunc: IN => Boolean
override def transform(df: DataFrame): DataFrame = {
transformSchema(df.schema, logging = true)
val transformUDF = udf[Boolean, IN](this.createFilterFunc)
df.filter(transformUDF(df($(inputCol))))
}
}
this code did not compile with an error:此代码未编译并出现错误:
No TypeTag available for IN
[error] val transformUDF = udf[Boolean, IN](this.createFilterFunc)
how do I let this work?我该如何让它工作?
I need it to work with some explicit defined type in inherit class such as我需要它在继承类中使用一些显式定义的类型,例如
class PriceFilter extends RuleFilter {
def createFilterFunc(val: Double) = val > 500
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.