how to take lines from a dataframe that are not in another dataframe using spark/scala

Question

I have a dataframe :

+++++++++++++++++++++++
|      Col1  |  col2    |
|+++++++++++++++++++++ |
|   A        |    A2   |
|   A        |    A2   |
|   B        |    b2
|   B        |    b2   |
|   C        |    c2   |
|   D        |    d2   |
|   E        |    e2   |
|   F        |    f2   |

And another dataframe

 +++++++++++++++++++++++
|      Col1  |  col2    |
|+++++++++++++++++++++ |
|   A        |    A2   |
|   B        |    b2   |
|   C        |    c2   |

I want have in result :

+++++++++++++++++++++++
|      Col1  |  col2    |
|+++++++++++++++++++++ |
|   D        |    d2   |
|   E        |    e2   |
|   F        |    f2   |

I do that :

df1.join(df2,Seq("col1","col2"),"left")

But doesn't work for me .

Any idea ? Thank you .

Answer 1

We can use .except or leftjoin for this case.

Example:

df.show()
//+----+----+
//|Col1|Col2|
//+----+----+
//|   A|  A2|
//|   A|  A2|
//|   B|  b2|
//|   B|  b2|
//|   C|  c2|
//|   D|  d2|
//|   E|  e2|
//|   F|  f2|
//+----+----+

df1.show()
//+----+----+
//|Col1|Col2|
//+----+----+
//|   A|  A2|
//|   B|  b2|
//|   C|  c2|
//+----+----+

df.except(df1).show()
//+----+----+
//|Col1|Col2|
//+----+----+
//|   E|  e2|
//|   F|  f2|
//|   D|  d2|
//+----+----+


df.alias("d1").join(df1.alias("d2"),
    (col("d1.Col1")===col("d2.Col1") &&(col("d1.Col2")===col("d2.Col2"))),"left").
filter(col("d2.Col2").isNull).
select("d1.*").
show()
//+----+----+
//|Col1|Col2|
//+----+----+
//|   D|  d2|
//|   E|  e2|
//|   F|  f2|
//+----+----+

Answer 2

You can use except on both the df.

scala> df1.except(df2).show
+----+----+
|Col1|col2|
+----+----+
|   E|  e2|
|   F|  f2|
|   D|  d2|
+----+----+