Connecting to multiple databases efficiently

Question

Referring to similar question :

Pattern for connecting to different databases using JDBC

I am using different Connection Strings/Drivers for each database.This is what I am doing, not very sure if it's the most efficient way to do it:

Create separate classes for each db's Connection with a getConnection(String URl,String userid,String password) method in it

In main class get connection object for DB1,DB2,DB3, open connections

Fetch data from DB1, write it to a flat file, repeat for DB2 and DB3

Close all three connections.

NOTE:I read about using Spring/Hibernate/DataSources/ConnectionPooling Dont know what shoud be the best option

Answer 1

The way I understand it is that you want your application to run some ( SELECT ?) queries on different databases and dump the results. I presume this is a part of a larger application since otherwise you would probably get results quicker by simply writing a command-line script that automates the client tools for the specific databases.

Hibernate , Data Sources (in the Java DataSource object sense) and Connection Pooling won't solve your problem - I guess it's the same for Spring but I don't know which part of Spring you're referring to. The reason for this is that they all are designed to abstract over a single (or a pool/collection of connections) to a single database - connection pooling simply allows you to keep a pool of ready-to-use (TCP) connections to a given database in order to improve performance, for example by avoiding connection and authentication overhead. Hibernate does the same in the sense that it abstracts a connection to a single database (and can use connection pooling for performance reasons on top of that).

I would suggest to maybe take a different approach to thinking about your problem:

Since you want to run some queries on some datasource and write the results to some destination, why don't you start your design this way: Come up with an interface/class DataExtractionTask that requires a database connection, a set of queries to run and some output stream. Instead of using java.sql.Connection directly you could choose some framework to make your life easier, there are heavy-weights like Hibernate and light-weights like jdbi . Then come up with code that establishes your database connection, decides which queries to run and the outputs to write to and feed all of that into your thought-out DataExtractionTask to run the logic of processing (orchestrating the individual parts).

Once you have the basic stuff in place you can add other features on top of it, you could make it configurable, you could choose to run multiple DataExtractionTask s in parallel instead of sequentially, et cetera.

This way you can generalize the processing logic and then focus on getting everything (database connections, query definitions, etc.) ready for processing. I realize that this is very broad-picture but maybe it makes things a bit easier.

Regarding efficiency: If you mean high performance (relative terms!), the best way would be what @Elliott Frisch wrote -- keeping it all in a single database that you connect to using a single connection pool.

Answer 2

You don't need to use separate classes just for connecting, just build up a util class which holds all the JDBC URLs and obtain a connection from it.

Besides that, you should consider using JPA instead, which you can do as well in Java SE as in Java EE. With that, you can abstract from the low level connection and define a named datasource. See for example this Oracle tutorial .