Oracle to SQL Server data transfer using spring boot

Question

I am looking for technical solution; To query data from one db and load it into a SQL Server database using java spring boot.

Mock query to get productNames which are updated between given time of 20 hours:

SELECT 
    productName, updatedtime FROM
products WHERE
    updatedtime BETWEEN '2018-03-26 00:00:01' AND '2018-03-26 19:59:59';

Here is the approach we followed.

1) Its long running Oracle query, which runs approximately 1 hours on business hours and it returns ~1Million records.

2) We have to insert/ dump this resultset into a SQL Server Table using JDBC.

3) As I know Oracle JDBC driver supports kind of streaming. When we iterate over ResultSet it loads only fetchSize rows into memory.

int currentRow = 1;
while (rs.next()) {
  // get your data from current row from Oracle database and accumulate in a batch
  if (currentRow++ % BATCH_SIZE == 0) {
    //insert whole accumulated batch into SqlServer database
  }
}

In this case we do not need to store all huge dataset from Oracle in memory. And we will insert into SqlServer by batches of BATCH_SIZE. The only thing is that we need to think where to do commit into SqlServer database.

4)Here is the bottleneck is query execution waiting time to get the data from oracle db, So I am planing to split the query into 10 equal parts such each query to give updatedtime between each hour as shown. so that execution time also get reduced to ~10min for each query. eg: SELECT productName, updatedtime FROM products WHERE updatedtime BETWEEN '2018-03-26 01:00:01' AND '2018-03-26 01:59:59';

5.For that I required 5 Oracle JDBC connections and 5 Sql server connection(to query the data and insert into db) to do its job independently. I am new to JDBC connection pooling How can I do the connection pooling and closing the connection if its not in use etc?

Please suggest if you have any other better approach to get the data from the data source quickly as real time data. Please suggest. Thanks in advance.

Answer 1

This is a typical use case from spring batch.

There you have the concept of ItemReader(from your source db) and ItemWriter(into your destination db).

You can define multiple datasource and you will have capabilities for reading in fixed fetch size(JdbcCursorItemReader for instance) and also to create grid for parallel execution.

With a quick search you can find many examples online relative to this kind of tasks.

I know I'm not posting the code relative to the concept but it will take me some time to prepare a decent example