处理海量数据

Question

I am working on an interface which does DB interaction on some system. As part of my work, I am supposed to query the source db, invoke some procedure, get the data in a reference cursor and populate the destination db.

As the data volume can be huge, I am using multi threading on the destination db to invoke the procedure. For ex, if the total number of entries that are supposed to loaded is 1 million, then on destination db, the procedure is invoked say 10 times with 100K records each.

This arrangement is working fine except when the data volume at the source db is huge (for ex more than 2 million entries). I have set around 20 GB of heap space for processing the record but my program is failing with heap memory error.

I want to know if there is a way to query the data from the source db in parallel mode (for ex, assuming, a total of 2 million records is fetched from the source stored procedure, my program should first fetch a subset of this record and then move on to next or something like that).

One of the solutions that I have proposed is to send the records in this manner though db side but I want to know if there is a better alternative. Please suggest

Answer 1

I found a solution to this. The BeanPropertyRowMapper class in Spring API needs to be extended and you need to override the mapRow method. The mapRow method is called when the data is ready to be fetched. You can apply some kind of batching mechanism at this stage. Please note that I posted the question as the data is fetched using the Stored procedure and the output comes in the form of a reference cursor.