如果第一列中存在第二列值，则从两列中选择值的Spark Sql查询

Question

Input-

col_a   col_b
 A        B
 D        B
 B        E
 C        A

I am trying to get output in following way using sparksql but I am unable to get desired output using NOT EXITS/Left Outer join. Please help me to get following output.

col_a    col_b
 A         B
 D         B
 C         A

I want to get values from both table if col_b value is present in col_a.

Answer 1

Supposing that your columns aren't too large, I would do something like this:

scala> val df = Seq(("A", "B"), ("D", "B"), ("B", "E"), ("C", "A")).toDF("col_a", "col_b")
df: org.apache.spark.sql.DataFrame = [col_a: string, col_b: string]

scala> df.show
+-----+-----+
|col_a|col_b|
+-----+-----+
|    A|    B|
|    D|    B|
|    B|    E|
|    C|    A|
+-----+-----+

scala> import org.apache.spark.sql.Row
import org.apache.spark.sql.Row

scala> import scala.collection.mutable.HashSet
import scala.collection.mutable.HashSet

scala> val col_a_vals = df.rdd.map{case Row(a: String, b: String) => a}.collect.toSeq
col_a_vals: Seq[String] = WrappedArray(A, D, B, C)

scala> val col_a_set = HashSet(col_a_vals :_*)
col_a_set: scala.collection.mutable.HashSet[String] = Set(B, C, D, A)

scala> val broad_set = sc.broadcast(col_a_set)
broad_set: org.apache.spark.broadcast.Broadcast[scala.collection.mutable.HashSet[String]] = Broadcast(56)

scala> val contains_col_a = udf((value: String) => broad_set.value.contains(value))
contains_col_a: org.apache.spark.sql.UserDefinedFunction = UserDefinedFunction(<function1>,BooleanType,List(StringType))

scala> df.filter(contains_col_a($"col_b")).show
+-----+-----+
|col_a|col_b|
+-----+-----+
|    A|    B|
|    D|    B|
|    C|    A|
+-----+-----+