How to process large log file in java

Question

I have 4 files and each one is 200 MB. I have created 4 threads and parallelly running 4 thread and each thread processing and adding in to Array blocking queue.

Some other thread is taking Array Blocking Queue and process and adding in to batch. The batch size is 5000 and executing batch and inserting records into database.But still its taking complete 4 files is around 6 mins to complete.

How increase performance in this case?

Answer 1

1) Make sure you have enough memory for queue+processor buffers+db buffers.

2) Batch size of 5k is a bit more than needed, in general you get up to speed in 100, not that iе makes much difference here though.

3) You can push data into oracle in multiple threads. Fetching sequences for ID fields population ahead, you'll be able to insert into 1 table in parallel, if you have not many indexes. Otherwise consider disabling/recalculating indexes, or insert into temporary table and then move everything into main one.

4) Take a look at oracle settings with fellow DB admin. Things like extend size/increase can change performance.