Memory leak on Tomcat 7 with Oracle JDBC drivers 12c - oracle.jdbc.driver thread failed to stop

Question

I have a web app deployed to Tomcat 7.0.54 that uses a datasource to connect to an Oracle 11g database. The datasource is configured in META-INF/context.xml and I've placed ojdbc7.jar in <tomcat-install-dir>/lib . I use a JNDI lookup to retrieve the datasource which I store in a singleton so that each DAO class can use it.

Everything works as expected, however when I undeploy the application (via Tomcat manager app) I see in the logs:

Oct 03, 2014 3:06:55 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/myapp] appears to have started a thread named [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak.
Oct 03, 2014 3:06:57 PM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/myapp]

When I debug I can see this thread gets created as soon as the database is accessed (via the datasource).

My datasource config:

<Context antiResourceLocking="false">
    <Resource name="jdbc/myapp" auth="Container" 
        type="javax.sql.DataSource" driverClassName="oracle.jdbc.OracleDriver" 
        maxActive="20" maxIdle="10" maxWait="-1"
        username="myuser" password="mypass"
        url="jdbc:oracle:thin:@myserver:1521:mysid"
        removeAbandoned="true" removeAbandonedTimeout="10" logAbandoned="true"
        validationQuery="SELECT 1 FROM DUAL"   
        testOnBorrow="true" testOnReturn="true" testWhileIdle="true"  
        timeBetweenEvictionRunsMillis="1800000" numTestsPerEvictionRun="3"  
        minEvictableIdleTimeMillis="1800000"
    />
</Context>

EDIT

Further investigation has revealed that the issue occurs whether or not the datasource is accessed during application (or servlet) initialisation.

In actual fact, the problematic thread is only created, and thus the issue only exists, when using the 12c versions of Oracle's JDBC drivers (either ojdbc6.jar or ojdbc7.jar).

If I revert to using the 11.2.0.4 version of ojdbc6.jar the thread is never created and the memory leak warning never appears.

Should I downgrade JDBC driver (as suggested in https://stackoverflow.com/a/9177263/4105953 )?

Answer 1

I found a lengthy discussion about the subject here . The conclusion is that it creates a "fixed size memory leak", ie subsequent redeploys will not increase the memory leak.
I do not have Oracle Support access, but the bug ID mentioned in the discussion is 16841748 (May 2013, might be solved now).

A possible workaround is to actually use the datasource once (get connection, do dummy query, close connection) when Tomcat is started via a custom servlet that is configured to "load-on-startup" in tomcat/conf/web.xml . This should start the Oracle driver thread(s) (see also the FAQ about driver threads) outside of the scope of the class-loader of your web-app, thus preventing the "fixed size memory leak".

Note that a similar issue exists for the MySQL JDBC driver but has a decent solution . Such a solution may exist for a recent version of the Oracle JDBC driver (I don't know).

Answer 2

This is normal, it happens when you hot deploy in Tomcat. It will not cause you any problems normally in production because you don't generally keep hot deploying updates in production, you just stop and restart the server instead.

Answer 3

This problem doesn't happen only with Oracle JDBC driver (oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser).

If any of the 3rd party dependency starts a daemon thread for carrying out some task, then while stopping the Tomcat logs Warning message for them also.

Also, this problem exists with Tomcat 8.5.X versions.

Solution: What i found as a solution is to get the Thread Group for the current thread and interrupt it. It will make sure that before shutting down the Tomcat all it's daemon threads are killed.

below code should be added to the "contextDestroyed" method ThreadGroup threadGroup = Thread.currentThread().getThreadGroup(); threadGroup.interrupt();