在制作商品地图时如何减少Spark的改组和时间？

Question

I am using spark to read a csv file like this :

x, y, z
x, y
x
x, y, c, f
x, z

I want to make a map of items vs their count. This is the code I wrote :

private def genItemMap[Item: ClassTag](data: RDD[Array[Item]],     partitioner: HashPartitioner): mutable.Map[Item, Long] = {
    val immutableFreqItemsMap = data.flatMap(t => t)
      .map(v => (v, 1L))
      .reduceByKey(partitioner, _ + _)
      .collectAsMap()

    val freqItemsMap = mutable.Map(immutableFreqItemsMap.toSeq: _*)
    freqItemsMap
  }

When I run it, it is taking a lot of time and shuffle space. Is there a way to reduce the time?

I have a 2 node cluster with 2 cores each and 8 partitions. The number of lines in the csv file are 170000.

Answer 1

If you just want to do an unique item count thing, then I suppose you can take the following approach.

val data: RDD[Array[Item]] = ???

val itemFrequency = data
  .flatMap(arr =>
    arr.map(item => (item, 1))
  )
  .reduceByKey(_ + _)

Do not provide any partitioner while reducing, otherwise it will cause re-shuffling. Just keep it with the partitioning it already had.

Also... do not collect the distributed data into a local in-memory object like a Map .