Spring Batch - FlatFileItemReader for different delimited files

Question

I have two different files, one delimited by pipe separator "|" and one by comma ",".

I am using Spring Batch to process these files using FlatFileItemReader. I dont want to have two readers and two writers for the files. Can I have somehow one generic FlatFileItemReader for both files?

Also the objects to which the files will be mapped are different.

Answer 1

You can inject a DelimitedLineTokenizer and you can set the delimiter as per requirement. you can make it generic using StepExecutionListener and need to override the beforeStep() method. You will set delimiter in StepExecution. When you parse the file which are "," seperated then stepExecution.getExecutionContext().putString("delimiter", ","); and when file is seperated with "|" then stepExecution.getExecutionContext().putString("delimiter", "|");

But you need to create two jobs. Need to specify listener accordingly.

You can see the example of above explain logic from Spring Batch on Walking Techie
Code for generic Reader:

@Bean
  @StepScope
  public FlatFileItemReader<Domain> reader(@Value("#{stepExecutionContext[delimiter]}") String delimiter) {
    FlatFileItemReader<Domain> reader = new FlatFileItemReader<>();
    reader.setResource(new ClassPathResource("sample-data.csv"));
    reader.setLineMapper(new DefaultLineMapper<Domain>() {{
      setLineTokenizer(new DelimitedLineTokenizer() {{
        setNames(new String[]{"id", "name"});
        setDelimiter(delimiter);
      }});
      setFieldSetMapper(new BeanWrapperFieldSetMapper<Domain>() {{
        setTargetType(Domain.class);
      }});
    }});
    return reader;
  }

You can find the many examples on spring batch in spring boot from Spring Batch Tutorial . You will find here all kind of problems related to spring batch.

Answer 2

You can inject a DelimitedLineTokenizer into the FlatFileItemReader and set the delimiter value to it. Relevant parts of the XML config are as follows

<bean id="pipeDelimitedFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
    ...
    <property name="lineMapper">
        <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
            <property name="lineTokenizer">
                <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                    ...
                    <property name="delimiter" value="|"/>
                </bean>
            </property>
            ...
        </bean>
    </property>
    ...
</bean>

You can eg similarly configure another, commaDelimitedFileItemReader bean (comma is actually the default delimiter value for DelimitedLineTokenizer)

Answer 3

I have created a sample spring batch programme in spring boot that will create two jobs one job will handle to read data from CSV file which is separated by the comma and other jobs to read data from CSV file which is separated by the pipe("|"). Both jobs are using the same common FlatFileItemReader to read data from CSV files and MongoItemWriter to write data into MongoDB.

You can find explanation and working code from Spring Batch Example in Spring Boot - CSV Files with different delimiter to Mongo Database