如何根据时间或特定数量批量处理事件

Question

I have an event system that I can subscribe to for when a specific object is changed. After receiving this event, I want to execute a task for this object.

It is possible that multiple objects are changed at the same time. Eg if I change 1000 objects I get 1000 events. The problem is that it takes way longer for the task I want to execute to process 1 objects 1000 times than 1000 objects 1 time. I cannot change the way the events are generated.

So what I thought about is to batch up these events when I receive them. Eg Collect 1000 Items in a Queue and the execute the task on all objects from the collected events. The problem is: what happens when only 999 objects are changed? Then my task is never executed. So I also want to drain the queue eg 5 seconds after the first object was inserted.

Is there any library for this specific task? Or do I have to build this myself with a Queue and some logic to do the things I want?

Answer 1

I'm almost sure that doesn't exist some specific lib for this, what I done once I needed a same strategy for events like you, was create a queue or a repository to store the events, and started a ScheduledExecutorService with a task running at a fixed rate, to consume the events, if there isn't events to consume I just skiped the execution. You can even put a verification in the store add method to see if the store has 1000 or more and hasn't been processed yet, so you can fire the task.