Result of Dataframe as a variable to another dataframe using Scala

Question

In my scenario, I need to get a result from a dataframe and use that result as a variable to another dataframe.

val newdate = spark.sqlContext.sql("select interval_startdt from FROMCOSMOS order by interval_startdt ASC limit 1")
                   .collectAsList.toString.replaceAll("[\\[\\]]","'")

RESULT : newdate: String = ''2014-06-27 00:00:00''

val finalresult = spark.sqlContext.sql("select * from Table2 where interval_startdt='$newdate'").show

The above one doesn't give me any values but when I insert the actual date it gives me the result.

Can anyone help me to solve this one.

Answer 1

I see a couple of problems with your query:

First, you're trying to use string interpolation ( '$newdate' ), but you didn't put s" in front of your sql string, so value of newdate is not in the query actually.

Second, as far as I understand, in Table2 , column interval_startdt is probably of type "date" or "timestamp". So you should probably rewrite your queue like "select * from Table2 where interval_startdt= to_timestamp('$newdate', "YYYY-MM-DD hh:mm:ss")"