如何使用Spring批处理有效且正确地将顺序活动加载到数据库？

Question

I'm currently working on a project loading a .dat file info to database. However, this .dat file contains not only data but also actions. The first field indicates the action of the records and all else are just data. Below are some examples records: A key1 key2 data1 data2 D key1 key2 data1 data2 C key1 key2 data1 data2

let, A=add, D=delete, C=update The file size is roughly 5GB. In this case, order of the records to be process does matter. Is it possible to use Spring batch to batch process this? A previous implementation done by my predecessor was to create 3 lists: addList, deleteList, updateList, and generate dml statements for them separately. Although very efficient when loading, but it doesn't work because doesn't matter which order you execute these lists, it might cause data inconsistency. So to keep the data valid, I believe these records have to be executed in order.

One solution I came up with is to set reader fetch-size to 1. But that seems to defeat the purpose of spring batch.

Are there better ways to do this? perhaps not spring batch but something else?

Answer 1

您是否调查过PatternMatchingCompositeLineTokenizer https://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/item/file/mapping/PatternMatchingCompositeLineMapper.html

Answer 2

You should be able to achieve this (while preserving records order) using a combination of:

• PatternMatchingCompositeLineMapper : to map items according to your patterns (A, D and C)
• ClassifierCompositeItemWriter : configured with a PatternMatchingClassifier . This will classify items according to their type and use the corresponding writer (you should have one writer per type) to do the action.