Spark dataframe udf No TypeTag available

Question

I am trying to extend spark ml pipeline model with a filter transformer, after

abstract class RuleFilter[IN, T <: RuleFilter[IN, T]]
    extends RuleTransformer with HasInputCol  {
  // def filterFuntion: String
  /** @group setParam */
  def setInputCol(value: String): T = set(inputCol, value).asInstanceOf[T]

  protected def createFilterFunc: IN => Boolean

  override def transform(df: DataFrame): DataFrame = {
    transformSchema(df.schema, logging = true)
    val transformUDF = udf[Boolean, IN](this.createFilterFunc)
    df.filter(transformUDF(df($(inputCol))))
  }
}

this code did not compile with an error:

 No TypeTag available for IN
[error]     val transformUDF = udf[Boolean, IN](this.createFilterFunc)

how do I let this work?

I need it to work with some explicit defined type in inherit class such as

class PriceFilter extends RuleFilter {
    def createFilterFunc(val: Double) = val > 500
}

Answer 1

You need to tell the compiler explicitly that you want a TypeTag for type In :

import scala.reflect.runtime.universe._
abstract class RuleFilter[In: TypeTag, T <: RuleFilter[In, T]]