java.lang.Long和scala.Long

Question

I don't know what happened in my code...

Logs are here.

[error] blahblah\SampleApp.scala:22:53: overloaded method value reduce with alternatives:
[error]   (func: org.apache.spark.api.java.function.ReduceFunction[java.lang.Long])java.lang.Long <and>
[error]   (func: (java.lang.Long, java.lang.Long) => java.lang.Long)java.lang.Long
[error]  cannot be applied to ((java.lang.Long, java.lang.Long) => scala.Long)
[error]     val sumHundred = sparkSession.range(start, end).reduce(_ + _)

When I ran this code in scala 2.11.12, spark 2.3.2 it works without any ERROR.
And same code in scala 2.12.7, spark 2.4.0 it doesn't works - what?

Anybody knows about this?

  private val (start, end) = (1, 101)

  def main(args: Array[String]): Unit = {
    val sumHundred = sparkSession.range(start, end).reduce(_ + _)
    logger.debug(f"Sum 1 to 100 = $sumHundred")
    close()
  }

There's a parent trait that builds sparkSession etc.

What I've tried:

1. Explicit declaration of type:
   private val (start: Long, end: Long) = ...
2. Similar things in reduce code.

What I know: Perfectly compatiable between scala.Long and java.lang.Long

Answer 1

It has nothing to do with Spark version. It is due to diffrences in Scala implementation between 2.11. and 2.12. You can see how code actual look like for line

val sumHundred = sparkSession.range(start, end).reduce(_ + _)

in Scala 2.11 (with scala.this.Predef.long2Long conversion)

val sumHundred: Long = sparkSession.range(start.toLong, end.toLong).reduce(((x$2: Long, x$3: Long) => scala.this.Predef.long2Long(scala.this.Predef.Long2long(x$2).+(scala.this.Predef.Long2long(x$3)))));

and Scala 2.12 (implicit conversions are not applied)

val <sumHundred: error>: <error> = sparkSession.range(start.toLong, end.toLong).<reduce: error>(((x$2: Long, x$3: Long) => x$2.$plus(x$3)));

Your code will compile if you add a flag scalacOptions += "-Xsource:2.11" .

This page has more info SAM conversion precedes implicits

PS. I would say the main source of amusement here is this SparkSession.range() method, which takes Scala Long parameters and returns Java Long value.

  def range(start: Long, end: Long): Dataset[java.lang.Long] = {

I would say it would be more consistent to choose one of them.