删除两个重复的行

Question

Good day colleagues. I have a big dataset (about 237 000 000 rows). There are a lot of columns. For example, I need to delete all duplicates with column names userId , VTS .

userId Vts moreColumn1 moreColumn2
10     150     2           3              -delete
11     160     1           6
10     150     0           1              -delete

I am bad with SQL. Have tried different variant from Internet, but it doesn't work.

UPDATE:

Ty for answers! I forgot to say i use java. There is my optimized code for java:

viewingDataset.groupBy("userId", "VTS")
                .count()
                .where("count = 1")
                .drop("count")
                .join(viewingDataset, JavaConversions.asScalaBuffer(asList("userId", "VTS")))

Answer 1

You can aggregate with count, filter the result and join back

df.groupBy("userId", "Vts").count
  .where($"count" === 1)
  .drop("count")
  .join(df, Seq("userId", "Vts"))

It is possible to get the same result with window functions, but it less robust if data is skewed and on average much more expensive.

Answer 2

You can achieve what you want with Window functions:

import org.apache.spark.sql.expressions.Window._

ds.withColumn("aux", count("*")
.over(Window.partitionBy($"userId", $"VTS")))
.where($"aux"===1)
.drop($"aux")

partitionBy will count how many elements are by partition according to the columns you are sending as paremeters (userId and VTS in your example). Then with the where clause we'll keep only the rows from partitions where the count is 1, ie the unique rows.

Result of the partitionBY clause

ds.withColumn("aux", count("*").over(Window.partitionBy($"userId", $"VTS"))).show

+-------+----+------------+------------+---+
| userId| VTS| moreColumn1| moreColumn2|aux|
+-------+----+------------+------------+---+
|     10| 150|           2|           3|  2|
|     10| 150|           0|           1|  2|
|     11| 160|           1|           6|  1|
+-------+----+------------+------------+---+

Final Result

+-------+----+------------+------------+
| userId| VTS| moreColumn1| moreColumn2|
+-------+----+------------+------------+
|     11| 160|           1|           6|
+-------+----+------------+------------+