Dead Connections Returned to JDBC Connection Pool - Glassfish 3.1.2.2

Question

I'm having an issue with the jdbc Connection Pool on glassfish handing out dead database connections. I'm running Glassfish 3.1.2.2 using jconn3 (com.sybase.jdbc3) to connect to Sybase 12.5. Our organization has a nightly reboot process during which time we restart the Sybase server. My issue manifests itself when an attempt to use a database connection during the reboot occurs. Here are the order of operations to produce my issue:

Sybase is down for restart.

Connection is requested from the pool.

DB operation fails as expected.

Connection is returned to the pool in a closed state.

Sybase is back up.

Connection is requested from the pool.

DB operation fails due to "Connection is already closed" exception.

Connection is returned to the pool

I've implemented a database recovery singleton that attempts to recover from this scenario. Any time a database exception occurs I make a jmx call to pause all queue's and execute a flushConnectionPool operation on the JDBC Connection Pool. If the database connection is still not up the process sets up a timer to retry in 10 minutes. While this process works, it's not without flaws.

I realize there's a setting on the pool so that you can require validation on the database connection prior to handing it out but I've shied away from this for performance reasons. My process performs approximately 5 million database transactions a day.

My question is, does anyone know of a way to avoid returning a dead connection back to the pool in the first place?

Answer 1

You've pretty well summed up your options. We had that problem, the midnight DB going down. For us, we turned on connection validation, but we don't have your transaction volume.

Glassfish offers a custom validation option, with which a class can be specified to do the validation.

By default, all the classes provided by Glassfish do (You'll see them offered as options in the console) is a SQL statement like this:

SELECT 1;

The syntax varies a bit between databases, SQL Server is uses '1', whereas for Postgres, it just uses 1. But the intent is the same.

The net is that it will cost you an extra DB hit every time you try to get a connection, but it's a really, really cheap hit. But still, it's a hit.

But you could implement your own version. It could do the check, say, every 10th request, or even less frequent. Roll a random number from 1 to N (N = 10, 20, 100...), if you get a '1', do the select (and fail, if it fails), otherwise return "true". But at the same time, configure it so that if you do detect an error, purge the entire pool. Clearly tweak this so you have a good chance of it happening when your db goes down at night (dunno how busy your system is at night) vs peak processing.

You could even "lower the odds" during peak processing. "if time between 6am and 6pm then odds = 1000 else odds = 100; if (random(odds) == 1) { do select... }"

A random option removes the need to maintain a thread safe counter.

In the end, it doesn't really matter, you just want a timely note that the DB is down so you can ask GF to abort the pool.

I can definitely see it thrashing a bit at the very beginning as the DB comes up, possibly refreshing the pool more than once, but that should be harmless.

Different ways you could play with that, but that's an avenue to consider.