Spark 數據幀 udf 沒有可用的 TypeTag

Question

我正在嘗試使用過濾器變壓器擴展spark ml管道模型，之后

abstract class RuleFilter[IN, T <: RuleFilter[IN, T]]
    extends RuleTransformer with HasInputCol  {
  // def filterFuntion: String
  /** @group setParam */
  def setInputCol(value: String): T = set(inputCol, value).asInstanceOf[T]

  protected def createFilterFunc: IN => Boolean

  override def transform(df: DataFrame): DataFrame = {
    transformSchema(df.schema, logging = true)
    val transformUDF = udf[Boolean, IN](this.createFilterFunc)
    df.filter(transformUDF(df($(inputCol))))
  }
}

此代碼未編譯並出現錯誤：

 No TypeTag available for IN
[error]     val transformUDF = udf[Boolean, IN](this.createFilterFunc)

我該如何讓它工作？

我需要它在繼承類中使用一些顯式定義的類型，例如

class PriceFilter extends RuleFilter {
    def createFilterFunc(val: Double) = val > 500
}

Answer 1

你需要明確地告訴編譯器你想要一個類型為In的TypeTag ：

import scala.reflect.runtime.universe._
abstract class RuleFilter[In: TypeTag, T <: RuleFilter[In, T]]