在spark中过滤KeyValueGrouped数据集

Question

I have a typed dataset of a custom class and use groupbykey method on it. You know that it results a KeyValueGroupedDataset. I want to filter this new dataset but there is no filter method for this type of dataset. So, My question is: How can I filter on this type of dataset? (Java solution is needed. spark version: 2.3.1).

sampleData:

"id":1,"fname":"Gale","lname":"Willmett","email":"gwillmett0@nhs.uk","gender":"Female"
"id":2,"fname":"Chantalle","lname":"Wilcher","email":"cwilcher1@blinklist.com","gender":"Female"
"id":3,"fname":"Polly","lname":"Grandisson","email":"pgrandisson2@linkedin.com","gender":"Female"
"id":3,"fname":"Moshe","lname":"Pink","email":"mpink3@twitter.com","gender":"Male"
"id":2,"fname":"Yorke","lname":"Ginnelly","email":"yginnelly4@apple.com","gender":"Male"

And What I did:

    Dataset<Person> peopleDS = spark.read().format("parquet").load("\path").as(Encoders.bean(Person.class));
    KeyValueGroupedDataset<String, Person> KVDS = peopleDS.groupByKey( (MapFunction<Person, String> ) f -> f.getGender() , Encoders.STRING());
//How Can I filter on KVDS's id field? 

Update1 (use of flatMapGroups):

Dataset<Person> persons = KVDS.flatMapGroups((FlatMapGroupsFunction <String,Person,Person>) (f,k) -> (Iterator<Person>) k ,  Encoders.bean(Person.class));

Update2 (use of MapGroups)

Dataset<Person> peopleMap = KVDS.mapGroups((MapGroupsFunction <String,Person,Person>) (f,g) -> {
        while (g.hasNext()) {
        //What can I do here?       
    }
},Encoders.bean(Person.Class); 

Update3 : I want to filter those groups that distinct of their ids is greater than 1. for example in below picture: I want just Female groups because distinct of their ids is greater that 1 (first field is id. Others are fname,lname,email and gender).

Update4: I did What I want with "RDD", but I want to do exactly this part of code with "Dataset":

List<Tuple2<String, Iterable<Person>>> f = PersonRDD
        .mapToPair(s -> new Tuple2<>(s.getGender(), s)).groupByKey()
        .filter(t -> ((Collection<Person>) t._2()).stream().mapToInt(e -> e.getId).distinct().count() > 1)
        .collect();

Answer 1

Grouping is used for aggregation functions, you can find functions like "agg" in "KeyValueGroupedDataset" class. If you apply aggregation function for ex. "count", you will get "Dataset", and "filter" function will be available.

"groupBy" without aggregation function looks strange, other function, for ex. "distinct" can be used.

Filtering example with "FlatMapGroupsFunction":

                .flatMapGroups(
                    (FlatMapGroupsFunction<String, Person, Person>) (f, k) -> {
                        List<Person> result = new ArrayList<>();
                        while (k.hasNext()) {
                            Person value = k.next();
                            // filter condition here
                            if (value != null) {
                                result.add(value);
                            }
                        }
                        return result.iterator();
                    },
                    Encoders.bean(Person.class))

Answer 2

Why don't you filter on id before grouping ? GroupByKey is an expensive action, it should be faster to filter first.

If you really want to group first, you may have to then use .flatMapGroups with identity function.

Not sure about java code but scala version would be something as follow:

peopleDS
.groupByKey(_.gender)
.mapGroups { case (gender, persons) => persons.filter(your condition) }

But again, you should filter first :). Specially since your ID field is already available before grouping.