枚举是Spark PairRDD引起问题的关键

Question

Some operation's on Spark PairRDD 's don't work correctly when the RDD's key is or contains an enum.

For example, the following piece of Spark code takes two weeks worth of weekdays and counts them, by weekday:

import java.time.DayOfWeek
val weekdays: Seq[(DayOfWeek, Int)] = DayOfWeek.values().map(dow => (dow, 1))
val numPartitions = 2 * weekdays.size
val result = sc
  .parallelize(weekdays ++ weekdays, numPartitions)
  .reduceByKey(_ + _)
  .collect
  .toSeq
println(result)

In the output, I'd expect every weekday (eg, MONDAY ) to have count 2, however, on my cluster, I get:

WrappedArray(
  (THURSDAY,1), (SATURDAY,1), (WEDNESDAY,2), (SATURDAY,1),
  (MONDAY,2), (TUESDAY,2), (THURSDAY,1), (FRIDAY,2), (SUNDAY,2)
)

If you run this on a cluster with a single node (or set numPartitions to 1), the result is correct (ie, all counts are 2).

Answer 1

Spark PairRDD 's operations like aggregateByKey() , reduceByKey() , combineByKey() take an optional argument to specify the Partitioner that Spark is to use. If you don't specify a partitioner explicitly, Spark's HashPartitioner is used, which calls a row's key's hashCode() method and uses it to assign the row to a partition. However, the hashCode() of an enum is not guaranteed to be the same on different JVMs processes – even if they run on the same Java version. As a consequence, Spark xyzByKey() operations don't work correctly.

In the above example, there are two pairs (THURSDAY, 1) in the input and each gets processed on a different executor. The example uses a HashPartitioner with 14 (= numPartitions ) partitions. Since (THURSDAY, 1).hashCode() % 14 produces different results on these two executors, these two rows get sent to different executors to be reduced, resulting in an incorrect result.

Bottomline: Don't use HashPartitioner with objects whose hashcode's aren't consistent over different JVM processes. In particular, the following objects aren't guaranteed to produce the same hashcode on different JVM processes:

• Java enum 's
• Scala sealed trait -based enum's:

sealed trait TraitEnum
object TEA extends TraitEnum
object TEB extends TraitEnum

• Scala abstract class -based enum's:

sealed abstract class AbstractClassEnum
object ACA extends AbstractClassEnum
object ACB extends AbstractClassEnum

• Any key that contains a nested object of one of the above types (and doesn't have a custom hashCode() implementation).

However, Scala case class -based enum's, have a consistent hashcode and are thus safe to use:

sealed case class CaseClassEnum(…) # “…" must be a non-empty list of parameters
object CCA extends CaseClassEnum(…)
object CCB extends CaseClassEnum(…)

Additional info:

• Blog post: http://dev.bizo.com/2014/02/beware-enums-in-spark.html
• Ticket on Spark with a proposal to catch this at compile/runtime