How to create a dataframe using the value of another dataframe

Question

I am getting suppId DataFrame using below code.

val suppId = sqlContext.sql("SELECT supp_id FROM supplier")

The DataFrame return single or multiple value.

Now I want to create a DataFrame using the value of supp_id from suppId DataFrame. But not understand, how to write this.

I have written below code. But the code is not working.

val nonFinalPE = sqlContext.sql("select * from pmt_expr) 
nonFinalPE.where("supp_id in suppId(supp_id)")

Answer 1

It took me a second to figure out what you're trying to do. But, it looks like you want rows from nonFinalPe that are also in suppId. You'd get this by doing an inner join of the two data frames which would look like below

val suppId = sqlContext.sql("SELECT supp_id FROM supplier")
val nonFinalPE = sqlContext.sql("select * from pmt_expr") 

val joinedDF = nonFinalPE.join(suppId, nonFinalPE("???") === suppId("supp_id"), "inner")