Packaging scala class on databricks (error: not found: value dbutils)

Question

Trying to make a package with a class

package x.y.Log


import scala.collection.mutable.ListBuffer
import org.apache.spark.sql.{DataFrame}
import org.apache.spark.sql.functions.{lit, explode, collect_list, struct}
import org.apache.spark.sql.types.{StructField, StructType}
import java.util.Calendar
import java.text.SimpleDateFormat
import org.apache.spark.sql.functions._
import spark.implicits._

class Log{
...
}

Everything runs fine on same notebook, but once I try to create package that I could use in other notebooks I get errors:

<notebook>:11: error: not found: object spark
import spark.implicits._
       ^
<notebook>:21: error: not found: value dbutils
  val notebookPath = dbutils.notebook.getContext().notebookPath.get
                     ^
<notebook>:22: error: not found: value dbutils
  val userName = dbutils.notebook.getContext.tags("user")
                 ^
<notebook>:23: error: not found: value dbutils
  val userId = dbutils.notebook.getContext.tags("userId")
               ^
<notebook>:41: error: not found: value spark
    var rawMeta =  spark.read.format("json").option("multiLine", true).load("/FileStore/tables/xxx.json")
                   ^
<notebook>:42: error: value $ is not a member of StringContext
    .filter($"Name".isin(readSources))

Anyone knows how to package this class with these libs?

Answer 1

Assuming you are running Spark 2.x, the statement import spark.implicits._ only works when you have SparkSession object in the scope. The object Implicits is defined inside the SparkSession object. This object extends the SQLImplicits from previous verisons of spark Link to SparkSession code on Github . You can check the link to verify

package x.y.Log


import scala.collection.mutable.ListBuffer
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.{lit, explode, collect_list, struct}
import org.apache.spark.sql.types.{StructField, StructType}
import java.util.Calendar
import java.text.SimpleDateFormat
import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession

class Log{

  val spark: SparkSession = SparkSession.builder.enableHiveSupport().getOrCreate()

  import spark.implicits._

  ...[rest of the code below]
}