I am trying to cogroup two RDDs in my function for some calculations. My code is below:
def myCogroupFunction(teachers: org.apache.spark.rdd.RDD[Teacher], students: org.apache.spark.rdd.RDD[Student]) {
val teacherPairs = teachers.map { t =>
val nameKey: String = t.getName()
(nameKey.toUpperCase(), t)
}
val studentPairs = students.map { s =>
val nameKey: String = s.getName()
(nameKey.toUpperCase(), s)
}
teacherPairs.cogroup(studentPairs)
}
However, I got errors saying:
value cogroup is not a member of org.apache.spark.rdd.RDD[(String, com.abc.Teacher)]
What did I do wrong here? Thanks a lot!
除非您使用的是旧版本的Spark,否则这应该起作用,在这种情况下,您需要通过以下方式导入隐式文件,以使其成为可能:
import SparkContext._
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.