How to run very long process in a java based web application?

Question

I need to run a very long process in a java based spring boot web application. The process consists of following steps.

1. Get details for about 3,00,000 users from the database.
2. Iterate over them.
3. Generate PDF file for each user using itext.
4. Save PDF file on the filesystem.
5. Update the database that the PDF file for the given user has been created.
6. Update the PDF path for the user in the database.

Now, this entire process can take lots of time. May be lots of hours or even may be days as it consist of creating pdf file for each user, then lots of db updates.

Also, I need this process to run in background so that the rest of the web application can run smoothly.

I am thinking of using Spring Batch or Messaging Queue. Haven't really used any of them, so not sure if they are proper frameworks for such kind of problem or which one of these two is best fit for the problem.

What is the ideal way to implement such kind of tasks?

Answer 1

If you can't name a requirement you expect to be satisfied by a framework / library you most likely won't need one...

Generating PDFs might need a lot of power, you might want to keep this background process away from your main web application on it's own machines.

If it's a simple java process it's usually easier to control and to move it around your environment.

To me this looks like a simple task for "plain" java - KISS. Or am I missing something?

I'd make sure the Finder used to fetch the users from the database is

• restartable, ie only fetches unprocessed users (in case you have to stop the processing because shit happens:-)
• runs in batches to keep the db round trips and load low
• is multi threadable ie can fetch users split into a given number of threads (userid mod numberOfThreads, assuming userId is evenly distributed) so you can add more machines / threads if necessary.

Answer 2

You should use spring batch for this process. When the user presses the button, you would launch the job asynchronously. It will then run in a separate thread and process all your records. The current status of the job can be obtained from the job repository. Spring batch is made for this type of processing.