c3p0 getConnection随着连接数的增加而挂起

Question

i"m running a web service on Heroku and using New Relic to monitor its performance. I'm using MySQL with Hibernate on top. My non default c3p0 settings are the following

hibernate.c3p0.maxStatementsPerConnection, 5
hibernate.c3p0.maxPoolSize, 35
hibernate.c3p0.minPoolSize, 5
hibernate.c3p0.initialPoolSize, 10
hibernate.c3p0.acquireIncrement, 10

Every single request to my web service hits the database at least a couple of times. After running a load test of about 200 requests/minute for 10min I see most of time is spent in

com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection

My guess it's waiting for a connection in the connection pool? The interesting part is as I increased

hibernate.c3p0.maxPoolSize, 40

the performance was worse off (longer wait time in the same getConnection call. During the test I can see that the max number of c3p0 connections is indeed open at the MySQL server (max connection set on MySQL's end is 300 , definitely not exhausted).

All of my database functions use the same format

public void executeTransaction( Session session, IGenericQuery<T> query, T entity )
{
    Transaction tx = null;

    try
    {
        tx = session.beginTransaction();

        query.execute( session, entity );

        tx.commit();
    }
    catch ( RuntimeException e )
    {
        try
        {
            tx.rollback();
        }
        catch ( RuntimeException e2 )
        {
        }

        throw e;
    }
    finally
    {
        if ( session != null )
        {
            session.close();
        }
    }
}

so I'm certain all sessions are closed, which should translate into connections closing. Why is the wait time more as I increase the max number of connections? It seems like performance increases from hibernate.c3p0.maxPoolSize, 25 to hibernate.c3p0.maxPoolSize, 30 , but drops after hibernate.c3p0.maxPoolSize, 35 . Are my values far off?

Thanks!

Answer 1

as a guess, i would try increasing numHelperThreads . you have a heavy load; maybe c3p0's administrative Threads are getting backed up. (You should be able to see this if you dump stack traces or use JMX to monitor c3p0. If you have enough helper threads, they should generally be idle(), wait()ing. If they are getting backed up, you'll see them mostly active and runnable, and by JMX you'll see tasks queued.)

an insufficiency of helper threads is consistent with your observed better-then-worse performance with maxPoolSize. initially you get what you want, more Connections at the ready, but then the helper Threads fail to keep up and adding more Connections just makes things worse.

given your settings, helper Threads shouldn't have too much work to do, UNLESS maxStatementsPerConnection is too small. if your app has more than 5 PreparedStatements that are run frequently, then you will end up churning through Statements and tying up helper Threads with Statement close() tasks. you might try making this value larger. it should be approximately (rounding up) the number of distinct PreparedStatements used on an ongoing basis by your application. (You can ignore single or very rarely used PreparedStatements, involved for example in setup or cleanup.) again, monitoring what helper threads are up to would give you information about whether this is the issue. (you'd see backed-up Statement close() tasks.)

so, things to try: increase numHelperThreads, increase maxStatementsPerConnection (or set it to zero, to turn off Statement caching entirely.)

good luck!