Tomcat + TomcatJDBC ServletContextListener open threads

Question

When undeploying an application from Tomcat there are threads left open.

org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/services] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.
org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/services] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak.

The application maintains a map of DataSources and runs a ScheduledExecutorService to update the map every 5 minutes.

@WebListener
public class DataSourceFactory implements ServletContextListener
{
    private static Map<String, DataSource> rdsDataSourceMap;
    private static ScheduledExecutorService scheduler;
    private static final long CONNECTION_MAP_REFRESH_INTERVAL = 5;

    @Override
    public void contextInitialized(ServletContextEvent event) 
    {
        scheduler = Executors.newSingleThreadScheduledExecutor();
        scheduler.scheduleAtFixedRate(new Runnable(){
            @Override
            public void run() {
                cacheDatasourceMap();
            }
        }, 0, CONNECTION_MAP_REFRESH_INTERVAL, TimeUnit.MINUTES);
    }

    @Override
    public void contextDestroyed(ServletContextEvent event) 
    {
        scheduler.shutdownNow();
        if (localPool != null) {
            localPool.close();
        }
        for (DataSource ds : rdsDataSourceMap.values()) {
            if (ds != null) {
                ds.close();
            }
        }
    }

    private void cacheDatasourceMap()
    {
        ...
    }

    ....
}

The DataSources are created using TomcatJDBC with the following parameters:

driver=com.mysql.jdbc.Driver
jmxEnabled=true
testWhileIdle=true
testOnBorrow=true
validationQuery=SELECT 1
testOnReturn=false
validationInterval=30000
timeBetweenEvictionRunsMillis=5000
maxActive=100
maxIdle=20
initialSize=10
maxWait=100000
removeAbandonedTimeout=60
minEvictableIdleTimeMillis=30000
minIdle=10
logAbandoned=true
removeAbandoned=true
jdbcInterceptors=org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer;org.apache.tomcat.jdbc.pool.interceptor.ResetAbandonedTimer;org.apache.tomcat.jdbc.pool.interceptor.SlowQueryReportJmx(threshold=10000)

UPDATE

After getting rid of the ScheduledExecutorService I am still seeing the Timer thread being left open. I have added a logging statement at the end of the contextDestroyed() and verified that it is getting passed closing the DataSources.

I have also verified that the MySQL Driver in Tomcat's lib and not in the WAR.

Answer 1

First of all, there is nothing that Tomcat can do about this, you are creating an Executor using Java SE. So an application server (a Java EE) cannot and should not be managing this ExecutorService you have created directly from Java SE. If you want to use a Java EE ExecutorService, consider using ManagedScheduledExecutorService which you will not need to worry about shutting down because it uses the app server's thread pool. With that out of the way, onto the question...

You are using shutdownNow() which is a "quick and dirty" way of shutting down an ExecutorService . If you want to bring your app down gently I would recommend using ExecutorService.shutdown() in combination with ExecutorService.awaitTermination() instead.

According to the doc, shutdownNow() makes no guarantees about what can actually be stopped.

This method does not wait for actively executing tasks to terminate.
...
There are no guarantees beyond best-effort attempts to stop processing actively executing tasks.

If you care about waiting for tasks to stop, you need to use awaitTermination() . The only thing shutdown() or shutdownNow() can do is call interrupt() , which may or may not actually stop the Thread. To await termination, do this:

executor.shutdown(); // or shutdownNow()
if (!executor.isTerminated())
    executor.awaitTermination(10, TimeUnit.SECONDS); // wait for up to 10s