Missing parameter type for expanded function for reduceLeft

Question

For the following method series of transforms the Intellij IDE seems to "understand" what is going on: it does not note any errors/warnings and correctly shows the data types:

  val recsWithNames = recsWithNamesAndCnts.map(_._1)
  .reduceLeft{ case (dfCum: DataFrame,dfNew: DataFrame) => dfCum.union(dfNew)}

Here is the type inference :

However it does not compile:

Error:(426, 68) missing parameter type for expanded function
The argument types of an anonymous function must be fully known. (SLS 8.5)
Expected type was: (?, org.apache.spark.sql.DataFrame) => ?
      val recsWithNames = recsWithNamesAndCnts.map(_._1).reduceLeft{ case (dfCum: DataFrame,dfNew: DataFrame) => dfCum.union(dfNew)}

This has been a bit of a pattern with reduceLeft so I typically end up converting to foldLeft . In particular the following does work:

  val dfs = recsWithNamesAndCnts.map(_._1)
  val recsWithNames = dfs.tail.foldLeft(dfs.head){ case (dfCum: DataFrame,dfNew: DataFrame) => dfCum.union(dfNew)}

But maybe in this case someone might provide some insight on the particular nuances of reduceLeft leading to this error.

Answer 1

reduceLeft and foldLeft have the following signatures:

def reduceLeft[B >: A](op: (B, A) => B): B

def foldLeft[B](z: B)(op: (B, A) => B): B

Both involve an op that takes a Function2 , hence both will work fine without the case match:

(dfCum: DataFrame, dfNew: DataFrame) => dfCum.union(dfNew)

// Or, shorthanded to: 

_ union _

On the other hand, case (dfCum, dfNew) => dfCum.union(dfNew) is a Function1 (in particular, a partial function of Tuple2). The compiler is able to interpret it and infer type B in foldLeft , but not in reduceLeft (my guess is due to B >: A ). It'll work if you help the compiler a litte:

reduceLeft[DataFrame]{ case (dfCum, dfNew) => dfCum.union(dfNew) }