简体   繁体   中英

How to filter data using flink according to time?

I have some data like this format:

(id, time, value)

The following simulation data are given(There may be duplicate data):

("a1", "2022-06-28 00:00:00", "0.23"), // The time interval is 15 minutes, and there is only 24-hour data of the day
("a1", "2022-06-28 00:15:00", "0.89"),
...
("a1", "2022-06-28 23:59:59", "0.11"),

("b1", "2022-06-28 00:00:00", "0.23"), 
("b1", "2022-06-28 00:15:00", "0.89"),
...
("b1", "2022-06-28 23:59:59", "0.11"),


("c1", "2022-06-28 00:00:00", "0.23"), 
("c1", "2022-06-28 00:15:00", "0.89"),
...
("c1", "2022-06-28 23:59:59", "0.11"),

Assuming now is 2022-06-28 16:00:00 , I want to calculate data 1h,45min,30min,15min ago and now.

The output should be like this:

("a1", "2022-06-28 15:00:00", "1"),
("a1", "2022-06-28 15:15:00", "1"),
("a1", "2022-06-28 15:30:00", "1"),
("a1", "2022-06-28 15:45:00", "1"),
("a1", "2022-06-28 16:00:00", "1"),
("b1", "2022-06-28 15:00:00", "1"),
("b1", "2022-06-28 15:15:00", "1"),
("b1", "2022-06-28 15:30:00", "1"),
("b1", "2022-06-28 15:45:00", "1"),
("b1", "2022-06-28 16:00:00", "1"),
("c1", "2022-06-28 15:00:00", "1"),
("c1", "2022-06-28 15:15:00", "1"),
("c1", "2022-06-28 15:30:00", "1"),
("c1", "2022-06-28 15:45:00", "1"),
("c1", "2022-06-28 16:00:00", "1"),

How to write the Flink program? It is better to write in Java or Scala. I would be much appreciated if you could show me some code snippet!

I would recommend to have a look at the Windowing Table Valued Functions from Flink. You can find the documentation and examples at https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/window-tvf/#tumble

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM