How to schedule jobs / tasks

Question

I do not know really what I am looking for and I am not asking for the best solution. I would like to ask for possible solutions to do what I am asking for. I am building an application using Spring Boot.

I have a database with words in it. I am using these words on a site to search for products and would like to run them over and over again with some time between each search.

So basically the application ask the database for all words that has not been searched for within a specific time (let say 5 minutes). The words that I get in my query response I send out to a KafkaQueue and they are then processed by workers. As soon as they have been used by a worker I make an update in the database with the current time.

So I make a search in the database like every 1 minute (or more often) to find the ones that has not been used for 5 min and then runs them again.

This gives me a lot of connections to the database and I was thinking if there is a better solution. The workers are also saving other data to other tables in the database to.

It is like 80 - 90 words that are turned over like every 5 minutes. I had a thought to pick some of them and then send them to like a scheduler to set them to run in the remaining of the time until 5 minutes have passed.

If I schedule like 20 tasks at the time will this effect the memory a lot?

Currently I am using postgresql but maybe this is not the best DB for this kind of execution? I will be able to remove and/or add new words so I do not know if it is possible to even use a in-memory database for the words.

Answer 1

Sounds like a scheduler from SpringBoot is what you need. Look into using the @Scheduled annotation. The following link should provide you with everything Scheduled Annotation .

Answer 2

I ended up using JobRunr which I was able to schedule different delays for each task I set up. So far it has been really smoth using it and it has been working out fine for me.

Scheduled annotation in Spring Boot works really fine to but not for me when I wanted to have more flexibility in setting up different delays for each task.