Scala —使用表達式求值將數據幀寫入csv文件

Question

這是使用表達式（字符串）的評估（Eval或類似方法）將數據幀寫入Scala中的csv文件。

 import org.apache.spark.sql.{SaveMode, SparkSession, SQLContext, Row, DataFrame, Column}
 import scala.reflect.runtime.universe._
 import scala.tools.reflect.ToolBox
 import scala.reflect.runtime.currentMirror

 val df = Seq(("a", "b", "c"), ("a1", "b1", "c1")).toDF("A", "B", "C")
 val df_write = """df.coalesce(1).write.option("delimiter", "\u001F").csv("file:///var/tmp/test")"""

 // This is one of my failed attempts - I have tried using the interpreter as well (code not shown here).    
 val tb = runtimeMirror(getClass.getClassLoader).mkToolBox()  
 toolbox.eval(toolbox.parse(df_write))

 Errors are: 
 object coalesce is not a member of package df ....

Answer 1

濕婆神，嘗試下面的代碼。 問題在於對象變量不在工具箱范圍內，因此無法評估表達式。

package com.mansoor.test

import org.apache.spark.sql.{DataFrame, SparkSession}

object Driver extends App {

  def evalCode[T](code: String): T = {
    import scala.tools.reflect.ToolBox
    import scala.reflect.runtime.{currentMirror => m}
    val toolbox = m.mkToolBox()
    toolbox.eval(toolbox.parse(code)).asInstanceOf[T]
  }

  val sparkSession: SparkSession = SparkSession.builder().appName("Test")
    .master("local[2]")
    .getOrCreate()

  import sparkSession.implicits._
  val df: DataFrame = Seq(("a", "b", "c"), ("a1", "b1", "c1")).toDF("A", "B", "C")

  val df_write =
    s"""
       |import com.mansoor.test.Driver._
       |
       |df.coalesce(1).write.option("delimiter", "\u001F").csv("file:///var/tmp/test")
       """.stripMargin

  evalCode[Unit](df_write)

  sparkSession.sparkContext.stop()
}