处理大量数据

Question

Question Goes like this.

Form one application I am getting approx 2,00,000 Encrypted values task

1. Read all Encrypted values in one Vo /list
2. Reformat it add header /trailers.
3. Dump this records to DB in one shot with header and trailer in seperated define coloums

I don't want to use any file in between processes What would be the best way to store 2,00,000 records list or something how to dump this record at one shot in db. is better to dived in chunks and use separate thread to work on it. please suggest some less time consuming solution for this.

I am using spring batch for this and this process will be one job.

Answer 1

Spring batch is made to do this type of operation. You will want a chunk tasklet. This type of tasklet uses a reader, an item processor, and writer. Also, this type of tasklet uses streaming, so you will never have all items in memory at one time.

I'm not sure of the incoming format of your data, but there are existing readers for pretty much any use-case. And if you can't find the type you need, you can create your own. You will then want to implement ItemProcessor to handle any modifications you need to do.

For writing, you can just use JdbcBatchItemWriter .

As for these headers/footers, I would need more details on this. If they are an aggregation of all the records, you will need to process them beforehand. You can put the end results into the ExecutionContext.

Answer 2

There are a couple of generic tricks to make bulk insertion go faster:

• Consider using the database's native bulk insert.

• Sort the records into ascending order on the primary key before you insert them.

• If you are inserting into an empty table, drop the secondary indexes first and then recreate them.

• Don't do it all in one database transaction.

I don't know how well these tricks translate to spring-batch ... but if they don't you could consider bypassing spring-batch and going directly to the database.