Method parallelize need a Seq. Each item in the seq will be one record.
def parallelize[T](seq: Seq[T],
numSlices: Int = defaultParallelism)
(implicit arg0: ClassTag[T]): RDD[T]
In your example, you need add a Seq to wrap the Tuple, and in this case the RDD only has ONE record
scala> val rdd = sc.parallelize(Seq(("100", List("5", "-4", "2", "NA", "-1"))))
rdd: org.apache.spark.rdd.RDD[(String, List[String])] = ParallelCollectionRDD[2] at parallelize at <console>:24
scala> rdd.count
res4: Long = 1
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.