How to start the same spring-batch job concurrently with different parameters?

Question

I want to start the same job configuration multiple times with different filename parameters, to import multiple files concurrently (each job one file).

@Configuration
public class MyJob {

    //the steps contain each a reader, processor and writer
    //reader is @JobScope
    @Bean
    public Job job(Step someStep, Step, someMoreStep) {
        System.out.println("registering job bean");
        return jobBuilderFactory.get(name)
            .start(someStep)
            .next(someMoreStep)
            .build();
    }
}

@Component
public class MyImporter {
    @Autowired
    private JobRegistry jobRegistry;

    @Autowired
    private JobLauncher launcher;

    private static final String FILENAME = "baseFilename";

    @Async
    public void run(String i) {
        p = new JobParametersBuilder();
        p.addDate("date", new Date());
        p.addString("filename", FILENAME + i + ".csv"); //injected via @Value jobParameter to job

        Job job = jobRegistry.getJob("getMyJob");
        launcher.run(job, p.toJobParameters());
    }
}

@Component
public class MyImportManager {
    @Autowired
    private MyImporter importer;

    //starts a  job multiple times with different "filename" parameters,
    //to simulate concurrent file imports with the same configuration job class
    public void start() {
        for (int i = 0; i < 4; i++) {
            importer.run(i);
        }
    }
}

@SpringBootApplication
@EnableBatchProcessing(modular = true)
@EnableAsync
public class MyConfig {

}

Problem: I can start multiple jobs, but they seem to use eg the same reader . If I run a single job i < 1 , everything is working fine. If I increase i , I'm getting weired inputs.

I'm using FlatFileItemReader where each value is read line by line. But when using concurrent imports (from different files), the input strings are often corrupt.

So I assume I'm not registering the jobs properly for concurrent imports? But why?

What is interesting: the line "registering job bean" is only printed once. But shouldn't it be printed as often as the async job is started?

Answer 1

FlatFileItemReader is not thread-safe, so your reader bean will need to be in "job" scope. Then you'll get one instance per job.

@Bean
@JobScope
public FlatFileItemReader<?> yourReaderBean(
        @Value("#{jobParameters[filename]}") String filename){
    FlatFileItemReader<?> itemReader = new FlatFileItemReader<?>();
    itemReader.setLineMapper(lineMapper());
    itemReader.setResource(new ClassPathResource(filename));
    return itemReader;
}