Spark Scala code does not compile when used with a encoder + by name

Question

This is a simplified version of my code that is failing to compile:

object CompileFail {
  import org.apache.spark.sql.{ Encoder, SparkSession }

  case class Foo(x: Int)

  def run(spark: SparkSession): Int => Int = {
    import spark.implicits._
    add(bar(encode[Foo]))
  }

  def bar(f: => Int): Int = 0

  def encode[A : Encoder]: Int = 0

  def add(a: Int)(b: Int) = a + b
}

It is failing with following non sensical message:

[error] Error while emitting CompileFail.scala
[error] value $u
[error] one error found
[error] Compilation failed

I am on Scala 2.12.15 and Spark 3.1.2 (but it fails on older Spark versions too).

Interesting to note:

1. If I change add(a)(b) to add(a, b) it compiles
2. If I change bar(f: => Int) to bar(f: Int) it compiles
3. If I change add(bar(encode[Foo])) to add(bar(encode[String])) it compiles

What am I doing wrong?

Answer 1

I don't think you are doing anything wrong, it's a bug in scala resolution. Only doing this simple change works.

object CompileFail {
  import org.apache.spark.sql.{ Encoder, SparkSession }

  case class Foo(x: Int)

  def run(spark: SparkSession): Int => Int = {
    import spark.implicits._
    val i = encode[Foo]
    add(bar(i))
  }

  def bar(f: => Int): Int = 0

  def encode[A : Encoder]: Int = 0

  def add(a: Int)(b: Int) = a + b
}

It looks like it has conflicts with the implicit resolution and the lambda conversion of the function.