Scala — Use evaluation of an expression to write dataframe to a csv file

Question

This is to use evaluation (Eval or something similar) of an expression (string) to write a dataframe to a csv file in Scala.

 import org.apache.spark.sql.{SaveMode, SparkSession, SQLContext, Row, DataFrame, Column}
 import scala.reflect.runtime.universe._
 import scala.tools.reflect.ToolBox
 import scala.reflect.runtime.currentMirror

 val df = Seq(("a", "b", "c"), ("a1", "b1", "c1")).toDF("A", "B", "C")
 val df_write = """df.coalesce(1).write.option("delimiter", "\u001F").csv("file:///var/tmp/test")"""

 // This is one of my failed attempts - I have tried using the interpreter as well (code not shown here).    
 val tb = runtimeMirror(getClass.getClassLoader).mkToolBox()  
 toolbox.eval(toolbox.parse(df_write))

 Errors are: 
 object coalesce is not a member of package df ....

Answer 1

Shiva, try the below code. The issue was that the object variables were not in scope for the toolbox and therefore it was unable to evaluate the expression.

package com.mansoor.test

import org.apache.spark.sql.{DataFrame, SparkSession}

object Driver extends App {

  def evalCode[T](code: String): T = {
    import scala.tools.reflect.ToolBox
    import scala.reflect.runtime.{currentMirror => m}
    val toolbox = m.mkToolBox()
    toolbox.eval(toolbox.parse(code)).asInstanceOf[T]
  }

  val sparkSession: SparkSession = SparkSession.builder().appName("Test")
    .master("local[2]")
    .getOrCreate()

  import sparkSession.implicits._
  val df: DataFrame = Seq(("a", "b", "c"), ("a1", "b1", "c1")).toDF("A", "B", "C")

  val df_write =
    s"""
       |import com.mansoor.test.Driver._
       |
       |df.coalesce(1).write.option("delimiter", "\u001F").csv("file:///var/tmp/test")
       """.stripMargin

  evalCode[Unit](df_write)

  sparkSession.sparkContext.stop()
}