reduceByKey type mismatch

Question

I got this list

res22: Array[(String, List[(String, Int)])] = Array((Door_182,List((IN,1), (IN,1))), (Door_89,List((IN,1), (IN,1), (IN,1))), (Door_180,List((IN,1), (IN,1), (IN,1), (IN,1))), (Door_83,List((IN,1), (IN,1), (IN,1))), (Door_177,List((IN,1), (IN,1))), (Door_23,List((IN,1), (IN,1))), (Door_128,List((IN,1), (IN,1))), (Door_34,List((IN,1), (IN,1))), (Door_18,List((IN,1), (IN,1))), (Door_32,List((IN,1))), (Door_76,List((IN,1), (IN,1), (IN,1))), (Door_87,List((IN,1), (IN,1), (IN,1))), (Door_197,List((IN,1), (IN,1))), (Door_133,List((IN,1), (IN,1))), (Door_119,List((IN,1), (IN,1))), (Door_113,List((IN,1), (IN,1), (IN,1), (IN,1), (IN,1))), (Door_155,List((IN,1), (IN,1), (IN,1), (IN,1), (IN,1))), (Door_168,List((IN,1), (IN,1), (IN,1))), (Door_115,List((IN,1), (IN,1))), (Door_9,List((IN,1), (IN,1))),...

I tried to sum the number of IN for each door with this:

scala> reduced.map(n => (n._1, n._2)).reduceByKey((v1,v2) => v1 + v2.toString).collect

I get this error:

<console>:32: error: type mismatch;
found   : List[(String, Int)]
required: String
          reduced.map(n => (n._1, n._2)).reduceByKey((v1,v2) => v1 + v2).collect
                                                                     ^

How I can solve this?

Answer 1

You can do it in two steps: for each key aggregate all lists together and then sum all values in each list:

val x = sc.parallelize(List(("Door_182",List(("IN",1), ("IN",1))), ("Door_89",List(("IN",1), ("IN",1), ("IN",1))), ("Door_180",List(("IN",1), ("IN",1), ("IN",1), ("IN",1))), ("Door_83",List(("IN",1), ("IN",1), ("IN",1))), ("Door_177",List(("IN",1), ("IN",1)))))
x.reduceByKey(_ ::: _)
  .map {
    case (door, list) => (door, list.foldLeft(0){
      case (count1, (in2, count2)) => count1 + count2
   })
  }.collect()
res3: Array[(String, Int)] = Array((Door_180,4), (Door_83,3), (Door_177,2), (Door_182,2), (Door_89,3))

Or in a single operation with aggregateByKey avoiding extra memory allocation:

x.aggregateByKey(0)(
  {
    case (count, list) => count + list.foldLeft(0){
       case (count1, (in2, count2)) => count1 + count2}
  }, 
  _ +_ 
  )
  .collect()