reduceByKey類型不匹配

Question

我有這個清單

res22: Array[(String, List[(String, Int)])] = Array((Door_182,List((IN,1), (IN,1))), (Door_89,List((IN,1), (IN,1), (IN,1))), (Door_180,List((IN,1), (IN,1), (IN,1), (IN,1))), (Door_83,List((IN,1), (IN,1), (IN,1))), (Door_177,List((IN,1), (IN,1))), (Door_23,List((IN,1), (IN,1))), (Door_128,List((IN,1), (IN,1))), (Door_34,List((IN,1), (IN,1))), (Door_18,List((IN,1), (IN,1))), (Door_32,List((IN,1))), (Door_76,List((IN,1), (IN,1), (IN,1))), (Door_87,List((IN,1), (IN,1), (IN,1))), (Door_197,List((IN,1), (IN,1))), (Door_133,List((IN,1), (IN,1))), (Door_119,List((IN,1), (IN,1))), (Door_113,List((IN,1), (IN,1), (IN,1), (IN,1), (IN,1))), (Door_155,List((IN,1), (IN,1), (IN,1), (IN,1), (IN,1))), (Door_168,List((IN,1), (IN,1), (IN,1))), (Door_115,List((IN,1), (IN,1))), (Door_9,List((IN,1), (IN,1))),...

我試圖用這個來總結每個門的IN數量：

scala> reduced.map(n => (n._1, n._2)).reduceByKey((v1,v2) => v1 + v2.toString).collect

我收到此錯誤：

<console>:32: error: type mismatch;
found   : List[(String, Int)]
required: String
          reduced.map(n => (n._1, n._2)).reduceByKey((v1,v2) => v1 + v2).collect
                                                                     ^

我該如何解決？

Answer 1

您可以分兩個步驟進行操作：對於每個鍵，將所有列表匯總在一起，然后對每個列表中的所有值求和：

val x = sc.parallelize(List(("Door_182",List(("IN",1), ("IN",1))), ("Door_89",List(("IN",1), ("IN",1), ("IN",1))), ("Door_180",List(("IN",1), ("IN",1), ("IN",1), ("IN",1))), ("Door_83",List(("IN",1), ("IN",1), ("IN",1))), ("Door_177",List(("IN",1), ("IN",1)))))
x.reduceByKey(_ ::: _)
  .map {
    case (door, list) => (door, list.foldLeft(0){
      case (count1, (in2, count2)) => count1 + count2
   })
  }.collect()
res3: Array[(String, Int)] = Array((Door_180,4), (Door_83,3), (Door_177,2), (Door_182,2), (Door_89,3))

或在帶有aggregateByKey的單個操作中避免額外的內存分配：

x.aggregateByKey(0)(
  {
    case (count, list) => count + list.foldLeft(0){
       case (count1, (in2, count2)) => count1 + count2}
  }, 
  _ +_ 
  )
  .collect()