如何根据 Spark Scala 中其他数据框中的多列匹配过滤 dataframe

Question

Say I have three dataframes as follows:

  val df1 = Seq(("steve","Run","Run"),("mike","Swim","Swim"),("bob","Fish","Fish")).toDF("name","sport1","sport2")
  val df2 = Seq(("chris","Bike","Bike"),("dave","Bike","Fish"),("kevin","Run","Run"),("anthony","Fish","Fish"),("liz","Swim","Fish")).toDF("name","sport1","sport2")

Here is tabular view:

I want to filter df2 to only the rows where sport1 and sport2 combinations are valid rows of df1. For example, since in df1, sport1 -> Run, sport2 -> Run is a valid row, it would return that as one of the rows from df2. It would not return sport1 -> Bike, sport2 -> Bike from df2 though. And it would not factor in what the 'name' column value is at all.

The expected result I'm looking for is the dataframe with the following data:

+-------+------+------+
|name   |sport1|sport2|
+-------+------+------+
|kevin  |Run   |Run   |
|anthony|Fish  |Fish  |
+-------+------+------+

Thanks and have a great day!

Answer 1

Try this,

val res = df3.intersect(df1).union(df3.intersect(df2))

+------+------+
|sport1|sport2|
+------+------+
|   Run|   Run|
|  Fish|  Fish|
|  Swim|  Fish|
+------+------+

Answer 2

To filter a dataframe based on multiple column matches in other dataframes, you can use join :

df2.join(df1.select("sport1", "sport2"), Seq("sport1", "sport2"))

As by default join is an inner join, you will keep only the lines where "sport1" and "sport2" are the same in the two dataframes. And as we use a list of columns Seq("sport1", "sport2") for the join condition, the columns "sport1" and "sport2" will not be duplicated

With your example's input data:

val df1 = Seq(("steve","Run","Run"),("mike","Swim","Swim"),("bob","Fish","Fish")).toDF("name","sport1","sport2")
val df2 = Seq(("chris","Bike","Bike"),("dave","Bike","Fish"),("kevin","Run","Run"),("anthony","Fish","Fish"),("liz","Swim","Fish")).toDF("name","sport1","sport2")

You get:

+------+------+-------+
|sport1|sport2|name   |
+------+------+-------+
|Run   |Run   |kevin  |
|Fish  |Fish  |anthony|
+------+------+-------+