Spring Batch FlatFileItemReader continue on incorrect number of tokens

Question

I'm using Spring Batch FlatFileItemReader to parse csv files. Every now and then I get an ill-formated line and the application completely crashes with:

Caused by: org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record: expected 11 actual 18

Is there any way to tell the FlatFileItemReader to continue (throw exception and continue or ignore and continue) without it completely exiting the application.

I'm guessing I may need to extend the FlatFileItemReader to make this happen as there does not appear to be any setting for this. Any suggestions on how best to proceed and make this happen?

Answer 1

You can configure SkipLogic for your batch jobs Here's a link to doc

Basically, if you are using Java Config to manage your Batch Job you can do something like this

stepBuilderFactory.get("step1")
                .<Person, Person>chunk(10)
                .reader(reader)
                .writer(writer)
                .processor(processor)
                .faultTolerant()
                .skipLimit(10)
                .skip(RuntimeException.class)
                .listener(skipListener) // if you want to add
                .build();

Answer 2

I was able to solve this by creating a class that extended the DefaultLineMapper that gets injected into the FlatFileItemReader .

I then overrode mapLine method like this:

@Override
public T mapLine(String line, int lineNumber) throws Exception {
    T t = null;

    try {
        t = super.mapLine(line, lineNumber);
    } catch (Exception e) {
        log.error("Unable to parse line number <<{}>> with line <<{}>>.", lineNumber, line);
    }

    return t;
}

Answer 3

I'm not sure if this is the same as your issue, but I was getting the same error when parsing a caret "^" delimited file that contained quote " characters.

The " character is intended to be used to to extend a field across line endings or to enclose a String which contains the delimiter . So my line that looks like this:

^3500 LCF Gas:  109" Wheelbase, Reg Cab^Chevrolet^3500 LCF Gas^109" Wheelbase, Reg Cab^L

would be parsed like it has only 2 fields:

1. 3500 LCF Gas: 109" Wheelbase, Reg Cab^Chevrolet^3500 LCF Gas^109" Wheelbase, Reg Cab
2. L

What I really wanted is 5 fields:

1. 3500 LCF Gas: 109" Wheelbase, Reg Cab
2. Chevrolet
3. 3500 LCF Gas
4. 109" Wheelbase, Reg Cab
5. L

My file didn't contain any ampersands so I changed the DelimitedLinetokenizer default quoteCharacter from " to & .

lineTokenizer.setQuoteCharacter('&');

Which fixed my issue that resulted in the same error when using Spring Batch.

Answer 4

我设法通过将DefaultLineMapper的属性“ lineTokenizer ”的属性“ strict ”设置为false来解决这个问题。