如何在Apache Spark中基于列的子集实现`except`？

Question

I am working with two schema in spark, table1 and table2 :

scala> table1.printSchema
root
 |-- user_id: long (nullable = true)
 |-- item_id: long (nullable = true)
 |-- value: double (nullable = true)

scala> table2.printSchema
root
 |-- item_id: long (nullable = true)
 |-- user_id: long (nullable = true)
 |-- value: double (nullable = true)

However, I have created these two from different sources. Basically each of them is holding a value information for ( user_id , item_id ) pair which is a floating point data type, and as such, prone to floating point errors. For example (1, 3, 4) in one table can be stored as (1, 3, 3.9998..) in another due to other calculations.

I need remove rows with ( user_id , item_id ) pair (guaranteed to be pair-wise unique) from table1 which are also present in table2 . Something like this:

scala> table1.except(table2)

However, there is no way to tell except when it should determine two rows to be same, which in this case is just ( user_id , item_id ). I need to disregard value for this.

How to do this using spark-sql?

Answer 1

Using a leftanti join would be a possible solution. This will remove rows from the left table that are present in the right table for the given key.

table1.join(table2, Seq("user_id", "item_id"), "leftanti")