在火花流中聚合后努力处理重复数据删除

Question

1.streaming data is coming from kafka 2.consuming through spark streaming 3.firstname,lastname,userid and membername ( using member names i am getting the member count eg mark,tyson,2,chris,lisa,iwanka - so here member count is 3

somehow i have to do the count its the requirmnt . but how can i remove deduplication after aggregation . its my concern

  val df2=df.select(firstname,lastname,membercount,userid)
  df2.writestream.format("console").start().awaitTermination

  or     
 df3.select("*").where("membercount >= 3").dropDuplication("userid")

 // this one is not working , but i need to do the same after
   count only so that in batches same user id will not come again.
   only first time entry i want.

Batch-1 output

  firstname         lastname          member-count            userid

  john              smith                   5                  1
  mark              boucher                 8                  2
  shawn              pollock                3                  3

---

batch-2 output

 firstname         lastname           member-count        userid

 john               smith             7  (prev.count 5)         1
shawn               pollock           12  (prev.count 8)        3
chris               jordan            6                         4

---

// but here i want batch -2 ---------output

1.The possibilty is the john smith ,shawn pollock count will increase again in next batches ,but i dont want to show or keep in output for next batches.

ie based on userid , i want entry for the one time only in batch output and neglect again the same user in batch output firstname lastname member-count userid chris jordan 6 4

Answer 1

Your question is hard to read, but as I understand you want a while loop with a condition?

var a = 10;
while(a < 20){
     println( "Value of a: " + a );
     a = a + 1;
  }

For example will print

value of a: 10
value of a: 11
value of a: 12
value of a: 13
value of a: 14
value of a: 15
value of a: 16
value of a: 17
value of a: 18
value of a: 19