在 Spark Scala API 中检查 Set 中元素是否存在时键入不匹配错误

Question

I have defined following function written in Scala, for finding out whether particular type of nodes(Project,Filter) is contained in a given set or not:

private val operators = Set(Project.getClass,Filter.getClass)    
def containsNode(plan: Seq[LogicalPlan]):Boolean=
    {
     for(p<- plan)
       {
         if(operators.contains(p.getClass))
           true
       }
  false
}

When running the code I am getting the following error for the above function:

Error:(182, 36) type mismatch; found : Class[T(in value $anonfun)] where type T(in value $anonfun) <: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan

required: Class[_ >: T(in value operators) with T <: Serializable]

 if(operators .contains(p.getClass))

I thought p.getClass would give same type of value that set operators expect. But it doesn't seem like it. I am new to Scala and Spark, so I am not sure what exactly is going on here. Can anyone help me fix this error?

Answer 1

In this case the problem is too precise type inference, you can fix it by explicitly saying operators is a set of any classes:

private val operators = Set[Class[_]](Project.getClass,Filter.getClass)

Though a second problem is that just using true like that won't return it as you probably intend. You can explicitly write return true but better to use exists which already contains the desired logic:

def containsNode(plan: Seq[LogicalPlan]): Boolean =
    plan.exists(p => operators.contains(p.getClass))

EDIT: a third possible problem is that Project in Project.getClass is not the class, but its companion object, you probably want classOf[Project] instead.