No implicits found for parameter evidence

Question

I have a line of code in a scala app that takes a dataframe with one column and two rows, and assigns them to variables start and end :

val Array(start, end) = datesInt.map(_.getInt(0)).collect()

This code works fine when run in a REPL, but when I try to put the same line in a scala object in Intellij, it inserts a grey (?: Encoder[Int]) before the .collect() statement, and show an inline error No implicits found for parameter evidence$6: Encoder[Int]

I'm pretty new to scala and I'm not sure how to resolve this.

Answer 1

Spark needs to know how to serialize JVM types to send them from workers to the master. In some cases they can be automatically generated and for some types there are explicit implementations written by Spark devs. In this case you can implicitly pass them. If your SparkSession is named spark then you miss following line:

import spark.implicits._

As you are new to Scala: implicits are parameters that you don't have to explicitly pass. In your example map function requires Encoder[Int] . By adding this import, it is going to be included in the scope and thus passed automatically to map function.

Check Scala documentation to learn more.