Spring batch FlatFileItemReader token error

Question

I have two csv files in my filesystem which I need to process with spring batch based on some request type.

File for request type: EOD (without headers)

12345,BBG
23232,BBG

Another file for request type: ANCIENT (without headers)

12345,BBG,20201115
23232,BBG,20201115

Both file have mandatory first two fields, id and source . ANCIENT file can optionally have 3rd and 4th field startDate and endDate

How can I create an FlatFileItemReader that will work for both files? Currently I have something like:

My ItemReader implementation to Read the csv file:

@Bean
@Scope(value = "step", proxyMode = ScopedProxyMode.INTERFACES)
public FlatFileItemReader<FixingDTO> fileReader(@Value("#{jobExecutionContext['type']}") String type) {
    SomeType type = forName(type);
    return new FlatFileItemReaderBuilder<MyDTO>()
            .name("fileReader")
            .resource(new ClassPathResource(MAP.get(type)))
            .delimited()
            .delimiter(",")
            .names("id", "source", "startDate", "endDate")
            .fieldSetMapper(myDTOMapper())
            .build();
}

Mapper For the Fields:

public class MyDTOFieldMapper implements FieldSetMapper<MyDTO> {

    @Override
    public MyDTO  mapFieldSet(FieldSet fieldSet) throws BindException {
        MyDTO dto = new MyDTO();

        dto.setId(fieldSet.readString("id"));
        dto.setSource(fieldSet.readString("source"));
        dto.setStartDate(formatDate(fieldSet.readString("startDate")));
        dto.setEndDate(formatDate(fieldSet.readString("endDate")));

        return dto;
    }

    private LocalDate formatDate(String dateString) {
        return LocalDate.parse(dateString, DateTimeFormatter.ofPattern("yyyyMMdd"));
    }
}

I get a Token Exception when running the job:

Caused by: org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record: expected 4 actual 2

I want to simply process whatever is in the line and assign to the object variables. StartDate and endDate can be null.

Answer 1

There is field strict (default value: true) inside of DelimitedLineTokenizer , that property exactly what you are looking for. You should set that field in false.

Description of behavior:

    /**
     * Public setter for the strict flag. If true (the default) then number of 
     * tokens in line must match the number of tokens defined 
     * (by {@link Range}, columns, etc.) in {@link LineTokenizer}. 
     * If false then lines with less tokens will be tolerated and padded with 
     * empty columns, and lines with more tokens will 
     * simply be truncated.
     * 
     * @param strict the strict flag to set
     */
    public void setStrict(boolean strict) {
        this.strict = strict;
    }

Unfortunately there is not nice way to set strict field in builder, otherwise we can set in following way:

@Bean
@Scope(value = "step", proxyMode = ScopedProxyMode.INTERFACES)
public FlatFileItemReader<FixingDTO> fileReader(@Value("#{jobExecutionContext['type']}") String type) {
    SomeType type = forName(type);
    return new FlatFileItemReaderBuilder<MyDTO>()
                .name("fileReader")
                .fieldSetMapper(myDTOMapper())
                .lineTokenizer(new DelimitedLineTokenizer() {
                    {
                        setDelimiter(",");
                        setNames("id", "source", "startDate", "endDate");
                        setStrict(false);
                    }
                })
                .build();
    }