How to create dynamic multiple jobs in spring batch

Question

I have created multiple jobs and i am running multiple jobs in a loop, my requirement to run these jobs sequentially,but item reader of all steps are running and then item writer of all jobs are running sequentially, is there is any solution so that when item reader and item writer of a job is complete then control goes to another job.

sample code is following :

public void jobprocess() throws Exception {
        List<Job> jobList = BatchJob.fetchAllJobs();
        for(Job job:jobList){            
            jobLauncher.run(job, params);
        }

    }
    
    public List<Job> fetchAllJobs() throws Exception {
            List<Object> list = service.findAll();
            List<Job> jobBuilderList = new ArrayList<>();
            int i = 0;
            for(Object obj:list){
    
                Step step = stepBuilderFactory.get("step"+i)
                    .<Student, Student>chunk(500)
                    .reader(ListItemReaderV2(obj))
                    .processor(Processor)
                    .writer(Writer)
                    .build();
    
                jobBuilderList.add(jobBuilderFactory.get("job"+i)
                    .incrementer(new RunIdIncrementer())
                    .listener(this)
                    .start(step)
                    .build());
                i++;
    
            }
            return jobBuilderList;
        }

Answer 1

You should construct a super job which encapsulates your multiple jobs as steps. By this way, you can ask Spring batch to run the steps (therefore your jobs) sequentially. In Spring batch there is a specific step ( JobStep ) which allow doing so:

public void  jobprocess() throws Exception {
    jobLauncher.run(superJob(), params);
}

public Job superJob(){
   List<Step> stepList = BatchJob.fetchAllJobs().stream()
                                           .map(j-> stepBuilderFactory.get(j.getName())
                                                        .job(j)
                                                        .build()
                                            ).collect(Collectors.toList());
   SimpleJobBuilder simpleJobBuilder=jobBuilderFactory.get("superJob")
                                                      .start(stepList.get(0));
   for(int i = 1; i < stepList.size() ; i++){
        simpleJobBuilder.next(stepList.get(i))
   }
   return simpleJobBuilder.build();
}

Sory for the old style loop coding: it is the simpliest way I see to exclude the first item of stepList . Notice that I used a default behaviour which injects the parent job params into each JobStep (cf. https://docs.spring.io/spring-batch/docs/4.2.x/api/org/springframework/batch/core/step/job/JobStep.html#setJobParametersExtractor-org.springframework.batch.core.step.job.JobParametersExtractor- )

Answer 2

After lot of experiments i found that this issue was due to use of ListItemReader() method in job class.

ListItemReader<Object> listItemReader(object obj) {
 return new ListItemReader<>(service.getList()); }

In above implementation listItemReader() method is calling even before job start and job is starting after execution of listItemReader() method.

Now i have created a separate reader class by implementing ItemReader interface and instead of returning ListItemReader now i am returning Only ItemReader. And by implementing this my jobs are running as expected.

Dummy Code of reader class is:

public class ReaderClass implements ItemReader<Object> {

    final private DataService dataService;

    final private entity entity;

    private List<Object> responseList;

    private int count = 0;

    @Override
    public Product read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
        if(responseList==null){
            responseList = DataService.fetchData(entity);
        }        
        Object obj = null;
        if (count < responseList.size()) {
            obj = responseList.get(count);
            count++;
        }
        return obj;
    }

}