Spring 批量 FlatFileItemWriter 从 Object 写入 csv

Question

I am using Spring batch and have an ItemWriter as follows:

public class MyItemWriter implements ItemWriter<Fixing> {

    private final FlatFileItemWriter<Fixing> writer;
    private final FileSystemResource resource;

    public MyItemWriter () {
        this.writer = new FlatFileItemWriter<>();
        this.resource = new FileSystemResource("target/output-teste.txt");
    }


    @Override
    public void write(List<? extends Fixing> items) throws Exception {

        this.writer.setResource(new FileSystemResource(resource.getFile()));
        this.writer.setLineAggregator(new PassThroughLineAggregator<>());
        this.writer.afterPropertiesSet();
        this.writer.open(new ExecutionContext());
        this.writer.write(items);
    }

    @AfterWrite
    private void close() {
        this.writer.close();
    }
}

When I run my spring batch job, the items are written to file as:

Fixing{id='123456', source='TEST', startDate=null, endDate=null}
Fixing{id='1234567', source='TEST', startDate=null, endDate=null}
Fixing{id='1234568', source='TEST', startDate=null, endDate=null}

1/ How can I write just the data so that the values are comma separated and where it is null, it is not written. So the target file should look like this:

123456,TEST
1234567,TEST
1234568,TEST

2/ Secondly, I am having an issue where only when I exit spring boot application, I am able to see the file get created. What I would like is once it has processed all the items and written, the file to be available without closing the spring boot application.

Answer 1

There are multiple options to write the csv file. Regarding second question writer flush will solve the issue.

1. https://howtodoinjava.com/spring-batch/flatfileitemwriter-write-to-csv-file/

2. We prefer to use OpenCSV with spring batch as we are getting more speed and control on huge file example snippet is below

   class DocumentWriter implements ItemWriter<BaseDTO>, Closeable { private static final Logger LOG = LoggerFactory.getLogger(StatementWriter.class); private ColumnPositionMappingStrategy<Statement> strategy; private static final String[] columns = new String[] { "csvcolumn1", "csvcolumn2", "csvcolumn3", "csvcolumn4", "csvcolumn5", "csvcolumn6", "csvcolumn7"}; private BufferedWriter writer; private StatefulBeanToCsv<Statement> beanToCsv; public DocumentWriter() throws Exception { strategy = new ColumnPositionMappingStrategy<Statement>(); strategy.setType(Statement.class); strategy.setColumnMapping(columns); filename = env.getProperty("globys.statement.cdf.path")+"-"+processCount+".dat"; File cdf = new File(filename); if(cdf.exists()){ writer = Files.newBufferedWriter(Paths.get(filename), StandardCharsets.UTF_8,StandardOpenOption.APPEND); }else{ writer = Files.newBufferedWriter(Paths.get(filename), StandardCharsets.UTF_8,StandardOpenOption.CREATE_NEW); } beanToCsv = new StatefulBeanToCsvBuilder<Statement>(writer).withQuotechar(CSVWriter.NO_QUOTE_CHARACTER).withMappingStrategy(strategy).withSeparator(',').build(); } @Override public void write(List<? extends BaseDTO> items) throws Exception { List<Statement> settlementList = new ArrayList<Statement>(); for (int i = 0; i < items.size(); i++) { BaseDTO baseDTO = items.get(i); settlementList.addAll(baseDTO.getStatementList()); } beanToCsv.write(settlementList); writer.flush(); } @PreDestroy @Override public void close() throws IOException { writer.close(); }

   }

Answer 2

Since you are using PassThroughLineAggregator which does item.toString() for writing the object, overriding the toString() function of classes extending Fixing.java should fix it.

Answer 3

1/ How can I write just the data so that the values are comma separated and where it is null, it is not written.

You need to provide a custom LineAggregator that filters out null fields.

2/ Secondly, I am having an issue where only when I exit spring boot application, I am able to see the file get created

This is probably because you are calling this.writer.open in the write method which is not correct. You need to make your item writer implement ItemStream and call this.writer.open and this this.writer.close respectively in ItemStream#open and ItemStream#close