从 Spark SQL 生成 SQL

Question

I have a Scala code with Spark DataFrame with a join of few frames and filter and dynamic part.

Is it possible to generate and write to log a classic SQL code to control the process?

        val target = List(
            ("0", "0", "Orange", "2020-03-10")
        ).toDF("id", "new_id", "name", "process_date")
    
    
    ....
    dynamic part of code
    ....
    
    increment.as("i")
            .join(target.as("t"), $"t.id" === $"i.id", "left")
            .select(selectFields: _*)

I want to get in logs something like this

    select field1, field2, ....
    from increment i join target t where t.id = i.id

Answer 1

Maybe you could try to write in the log itself just after the query's execution.

Something like this example:

import org.apache.log4j.{Level, Logger}

Logger.getRootLogger.setLevel(Level.ERROR)
// your code here
increment.as("i")
        .join(target.as("t"), $"t.id" === $"i.id", "left")
        .select(selectFields: _*)
Logger.getRootLogger.error("""SELECT column1, column2, column3 FROM table JOIN table1 ON(id = id1) WHERE column3 = .....GROUP BY ....""")