Spark Java API，数据集操作？

Question

I'm new spark Java API. My dataset contains two columns (account, Lib) . I want to display accounts having differents lib. In fact my dataset is something like this. ds1

 +---------+------------+
    |  account|    Lib     |
    +---------+------------+
    | 222222  |  bbbb      |
    | 222222  |  bbbb      |
    | 222222  |  bbbb      |
    |         |            |
    | 333333  |  aaaa      |
    | 333333  |  bbbb      |
    | 333333  |  cccc      |
    |         |            |
    | 444444  |  dddd      |
    | 444444  |  dddd      |
    | 444444  |  dddd      |
    |         |            |
    | 555555  |  vvvv      |
    | 555555  |  hhhh      |
    | 555555  |  vvvv      |

I want to get ds2 like this:

+---------+------------+
|  account|    Lib     |
+---------+------------+
|         |            |
| 333333  |  aaaa      |
| 333333  |  bbbb      |
| 333333  |  cccc      |
|         |            |
| 555555  |  vvvv      |
| 555555  |  hhhh      |

Answer 1

If groups are small you can use window functions:

import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window


df
  .withColumn("cnt", approx_count_distinct("Lib").over(Window.partitionBy("account")).alias("cnt"))
  .where(col("cnt") > 1)

If groups are large:

df.join(
  df
   .groupBy("account")
   .agg(countDistinct("Lib").alias("cnt")).where(col("cnt") > 1),
  Seq("account"),
  "leftsemi"
)