How to read multiple files in chunks using Spring-Batch?

Question

I'm using Spring-Batch to read csv files sequentially with MultiResourceItemReader .

I want to create a reader that

• reads the chunksize from file 1
• reads the chunksize from file 2
• compare both what has been read and create some kind of "patch" object
• write the patch object to database

Now the problem with MultiResourceItemReader is that it will first read the full file1 in chunks, and when the file is finished, it will continue with file2.

How can I create batch steps that will switch between the files based on the chunksize?

Answer 1

You're going to need to create a custom reader to address what you're attempting. You can use the FlatFileItemReader under the hood for the actual file reading, but the logic of reading from two files at once you'll have to orchestrate yourself. Just coding off the top of my head, I'd expect something like this:

public class MultiFileReader implements ItemReader<SomeObject> {

    private List<ItemStreamReader> readers;

    public SomeObject read() {
        SomeObject domainObject = new SomeObject();

        for(ItemStreamReader curReader : readers) {
            domainObject.add(curReader.read());
        }

        return domainObject;
    }
}

Answer 2

you could use something like

    @Bean
    public MultiResourceItemReader<Company> readerCompany() throws IOException {

        DelimitedLineTokenizer dlt = new DelimitedLineTokenizer();
        dlt.setDelimiter("^");
        dlt.setNames("name", "cui", "code", "euid", "companyState", "address");
        dlt.setStrict(false);

        return new MultiResourceItemReaderBuilder<Company>()
                .name("readerCompany")
                .resources(inputCompanyResources)
                .delegate(new FlatFileItemReaderBuilder<Company>()
                        .name("getCompanyStatusReader")
                        .fieldSetMapper(new FieldSetMapper<Company>() {
                            @Override
                            public Company mapFieldSet(FieldSet fieldSet) throws BindException {
                                return Company.builder()
                                        .name(fieldSet.readString("name"))
                                        .localId(fieldSet.readString("cui"))
                                        .code(fieldSet.readString("code"))
                                        .companyStatus(readCompanyStatuses(fieldSet.readString("companyState")))
                                        .address(fieldSet.readString("address"))
                                        .internationalId(fieldSet.readString("euid"))
                                        .build();
                            }
                        })
                        .linesToSkip(1)
                        .lineTokenizer(dlt)
                        .build())
                .build();
    }