Spring Batch - Is there a way to commit data even if the chunk raise some exception?

Question

I have a process that read from a queue, process and write into DB. Even if the process fails, I have to store in DB. But the Spring Batch steps are transactional and always rollback the changes. So, is there a way to commit data even if the chunk raise some exception?

EDIT I:

I tried with Tasklet but getting the same behaviour.

Thanks in advance.

Answer 1

During configuring a step, you can use noRollback() to configure a list of exception that will not cause rollback. Any exceptions that are the subclass of configured exception will not rollback. That means if you simply want to never rollback, set it as Exception which is the parent of all exceptions.

An example can be found from the docs :

@Bean
public Step step1() {
    return this.stepBuilderFactory.get("step1")
                .<String, String>chunk(2)
                .reader(itemReader())
                .writer(itemWriter())
                .faultTolerant()
                .noRollback(Exception.class)
                .build();
}

Answer 2

I tried to use noRollback() as Ken Chan suggested but did'nt work. Also try to put the specific exceptions, but it keeps doing rololback.

The conditional flow is at Step level no item level, so it doesn't help me. Also tried with Listeners but the documentation said:

This listener is designed to work around the lifecycle of an item. This means that each method should be called once within the lifecycle of an item and in fault tolerant scenarios, any transactional work that is done in one of these methods would be rolled back and not re-applied. Because of this, it is recommended to not perform any logic using this listener that participates in a transaction.

I solved my problem using Tasklet insted of chunked oriented solution and adding an @Transactional to the execute method of the Tasklet.

@Transactional(propagation = Propagation.REQUIRES_NEW, isolation = Isolation.SERIALIZABLE, noRollbackFor = {
        ErrorInternoServidorException.class, SolicitudIncorrectaException.class,
        RegistroNoEncontradoException.class, SolicitudEventoObjetaException.class,
        SolicitudEventoValidaException.class, MimCargueSolicitudException.class, ConflictException.class,
        UnauthorizedException.class, ForbiddenException.class }) 
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {

Spring Batch chunked oriented solution is wrapped in a Tasklet with his own transaction, so I tried to create a new one with my own rules.

Thanks to everyone for your replies. I learned a lot.

Answer 3

One way you could write your JOB to commit all your data even when exceptions are raised in your processing is to use a SkipPolicy and write your data to the destination DB there.

One of the main benefits of SkipPolicy is to log the data that caused and exception through the processing, and the logging part could even be inserting the record in a temporary Table in you DB.

public class FileVerificationSkipper implements SkipPolicy {

    private static final int MAX_VALUES_TO_SKIP = 1000;
    private JdbcTemplate jdbcTemplate;

    public FileVerificationSkipper(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    @Override
    public boolean shouldSkip(Throwable exception, int skipCount) throws SkipLimitExceededException {
        if (exception instanceof FlatFileParseException && skipCount <= MAX_VALUES_TO_SKIP) {
            jdbcTemplate.update("INSERT INTO YourTable(column1, column2) VALUES(?,?)");
            return true;
        } else {
            return false;
        }
    }
}

Hope this helps.