Multiple readers in single job in single step using spring batch

Question

I am newbie in spring batch. I have a use case in which, I've to read files from a specific folder and write those files in to DB.

For example, I've a files In folder like this

-company_group | -my_company_group.json -my_company_group_alternate_id.json -sg_company_group.json -sg_company_group_alternate_id.json

Note: sg = Singapore, my=Malaysia

Now, I want to read these files In the following order.

• SG files should be read first than my files.
• for each country alternate file should come first.

For example, sg_company_group_alternate_id.json sg_company_group.json

And same for the MY files

Currently, I'm reading all files by writing custom MultiResourcePartitioner and sorting my files order in the way which I mentioned above.

There will be 1 writer and reader for 1 file.

There will be 1 job.

Now, the problem Is I've a step in which I've a custom partitioner which I mentioned above it gets all file sort it but it goes in to only 1 reader. It will go through one reader for all files. I want multiple readers for all files.

I mean to say, in 1 job I've a step which loads all files. Now in this step, 1 file get read, write in db repeat for other file in same step.

As per my understanding spring batch do not allow multiple readers in 1 step.

Is there any workaround?

Thanks.

Answer 1

I would recommend to create a job instance per file, meaning pass the file as an identifying job parameter. This has at least two major benefits:

• Scaling: you can run multiple jobs in parallel, each job processing a different file
• Fault-tolerance: if a job fails, you only restart the failed job, without affecting other files