How to write Join and where in Spark DataFrame (Converting SQL to DataFrame)

Question

I need to write SQL Query into DataFrame

SQL Query

A_join_Deals = sqlContext.sql("SELECT * FROM A_transactions LEFT 
JOIN Deals ON (Deals.device = A_transactions.device_id) WHERE 
A_transactions.device_id IS NOT NULL AND A_transactions.device_id != '' AND 
A_transactions.advertiser_app_object_id = '%s'"%(adv_id))

Code written up to now

val A_join_Deals = Deals.join(A_transactions,Deals("device") === A_transactions("device_id"),"left")

Now I am stuck how to write the where Clause.

Any Suggestion or Help is Highly appreciated.

Answer 1

you can try as below

val A_join_Deals = Deals.join(A_transactions,Deals("device") === A_transactions("device_id"),"left")
      .where(A_transactions("device_id").isNotNull && A_transactions("device_id") =!= "" && A_transactions("advertiser_app_object_id") === s"${adv_id}")

Answer 2

Here is what you can do

where(A_transactions("device_id").isNotNull &&
   A_transactions("device_id") =!= "" && 
   A_transactions("advertiser_app_object_id") === s"${adv_id}%")

This works if adv_id is a variable and not Column

Hope this helps!