create a register dynamic dataframe as temptable in spark

Question

I am trying to registerTemptables, from dynamic dataframes.

I am getting the output as a string., i am not sure if there is a way to execute dataframe or convert a string to dataframe so that the temptable can be created.

Here are the steps to replicate this issue :

import org.apache.spark.sql._

val contact_df = sc.makeRDD(1 to 5).map(i => (i, i * i)).toDF("value", "square")
val acct_df = sc.makeRDD(1 to 5).map(i => (i, i / i)).toDF("value", "devide")

val dataframeJoins = Array( 
  Row("x","","",""  ,"Y","",1,"contact_hotline_df","contact_df","acct_nbr","hotline_df","tm49_acct_nbr"),
  Row("x","","","","Y","",2,"contact_hotline_acct_df","acct_df","tm06_acct_nbr"  ,"contact_hotline_df","acct_nbr")
)

val dfJoinbroadcast = sc.broadcast(dataframeJoins)

val DFJoins1 = for ( row <- dfJoinbroadcast.value ) yield {  
  (row(8)+".registerTempTable(\""+row(8)+"\")" )
}

for (rows <- 0 until DFJoins1.size ){
  println(DFJoins1(rows) )
  DFJoins1(rows)
}

Here is the output of the above for loop :

contact_df.registerTempTable("contact_df")
acct_df.registerTempTable("acct_df")

I am not getting any error. But the table is not getting created.

When i say sqlContext.sql("select * from contact_df") i am getting an error that table is not created.

Is there a way to convert string to a dataframe and execute the dataframe to create temptable.

Please suggest.

Thanks, Sreehari

Answer 1

Your code concatenates the strings and prints the result, that's it. The registerTempTable method is not being called, that's why you cant use it in the SQL query. Try to do this:

// assuming we have this string to object mapping
val tableNameToDf = Map("contact_df" -> contact_df, "acct_df" -> acct_df)

you could restructure your for loop into something like:

val dfJoins = for (row <- dfJoinbroadcast.value) yield {
   val wannabeTable = row(8)
   tableNameToRdd(wannabeTable).createOrReplaceTempView(wannabeTable)
   wannabeTableName
}