在scala中将两个RDD分组

Question

I am trying to cogroup two RDDs in my function for some calculations. My code is below:

def myCogroupFunction(teachers: org.apache.spark.rdd.RDD[Teacher], students: org.apache.spark.rdd.RDD[Student]) {

    val teacherPairs = teachers.map { t =>
      val nameKey: String = t.getName()
      (nameKey.toUpperCase(), t)
    }

    val studentPairs = students.map { s =>
      val nameKey: String = s.getName()
      (nameKey.toUpperCase(), s)
    }

    teacherPairs.cogroup(studentPairs)
  }

---

However, I got errors saying:

value cogroup is not a member of org.apache.spark.rdd.RDD[(String, com.abc.Teacher)]

What did I do wrong here? Thanks a lot!

Answer 1

除非您使用的是旧版本的Spark，否则这应该起作用，在这种情况下，您需要通过以下方式导入隐式文件，以使其成为可能：

import SparkContext._