异步Spring Batch。 分两步创建工作

Question

I have a requirement to call a webservice multiple times and pass data that is created using data from multiple Tables

UI -> Controller -> Service -> ( fetch data ( Table1, table2) and run some validation) 1. If validation is failed - return error message and stop. 2. If validation is passed - call JobLauncher and return "Task launched" message.

In The asynchronous job , I think of following 2 steps.

1. BulkInsertStep

   • I need to call DB to query 2 more tables ( Table3, table4) and create a large dataset, from earlier fetched Table1, Table2 and current fetched table3, table4

   • Insert created dataset into table5

2. WebServiceStep ( AysncTaskExecutor)

   • Reader: Query from Table5
   • Processor: Call Webservice for each row.
   • Writer: update Table5

I dont know if BulkInsertStep qualifies to be step in BatchJob or not. Basically its a for loop in for loop in for loop to create a List of DTO objects. What goes to reader, processor or writer is little confusing..

Avoiding BulkInsertStep avoids bulk insert and also reading in WebServiceStep , but if later user wants to rerun failed webservice records, it will be hard to figure out what permutations has already been processed and what permutations needs to be processed.

Please suggest a design or another way to achieve the same thing. Requirements:

1. Webservice calls for large data takes some time. So UI cannot wait for response.
2. User can rerun task, which processes failed tasks.

Answer 1

I dont know if BulkInsertStep qualifies to be step in BatchJob or not.

I think you definetely need to have this as Reader-Processor-Writer . While it is a bit complicated to write it pays off to you with such abilities as Restartability and Failure tolerance. I was also thinking before that Reader-Processor-Writer is an overkill, but at the end it also leads to better structured and maintainable code.

Avoiding BulkInsertStep avoids bulk insert and also reading in WebServiceStep, but if later user wants to rerun failed webservice records, it will be hard to figure out what permutations has already been processed and what permutations needs to be processed.

I've done something similar before. I would recommend here to store after every call to your WebService the result in the database. This is how table could look like:

- requestId 
- requestURL
- method (GET/POST/PUT/etc.) 
- requestBody (CLOB)
- requestHeaders 
- responseCode 
- responseBody (CLOB)
- responseHeaders

With this table I could be sure that there will be no duplicate WebService calls. If the batch job fails, I could simply restart Job on this step and continue processing starting with uncompleted calls.

For this you will need to have additional step to convert your data into requests (to prepare them), and then again using reader-processor-writer do WebService calls.

Webservice calls for large data takes some time. So UI cannot wait for response.

I assume you will start anyways whole job in separate thread. You need to configure your job launcher that it will use ThreadPoolTaskExecutor . Then job will run asyncronously.

User can rerun task, which processes failed tasks.

This is simply achieved. You have two options :

• by executing Failed job using jobOperator . Here you will need id of the failed job.
• by starting job with same identifying parameters. This is what you need to think upfront about. What are the parameters which uniquely identify your job? I recommend here to read official documentation: https://docs.spring.io/spring-batch/trunk/reference/html/domain.html#domainJobParameters