Improving performance of database tests using Spring 3.1, Hibernate 4.1, Dbunit, etc

Question

I'm currently starting a new project, and I've got around 190 repository tests. One thing I've noticed - and I am not entirely sure why this happening - is that the integration tests against HSQLDB (2.2.8) are running a lot slower than I think they should be.

I think I've tracked the bottleneck to the insertion of data before each test. For most tests, it ranges from .15 to .38 seconds just to setup the database. This is unacceptable. I would have imagined that an in-memory database would be much faster :(

Here is the database test class that all of my repository tests extend from:

@ContextConfiguration(locations = {"classpath:applicationContext.xml"})
@RunWith(SpringJUnit4ClassRunner.class)
@TransactionConfiguration(defaultRollback=true)
@Transactional
public abstract class DatabaseTest {

    public static final String TEST_RESOURCES = "src/test/resources/";

    @Autowired
    protected SessionFactory sessionFactory;

    @Autowired
    protected UserRepository userRepository;

    @Autowired
    protected DataSource dataSource;

    protected IDatabaseTester databaseTester;

    protected Map<String, Object> jdbcMap;
    protected JdbcTemplate jdbcTemplate;

    @PostConstruct
    public void initialize() throws SQLException, IOException, DataSetException {
        jdbcTemplate = new JdbcTemplate(dataSource);

        setupHsqlDb();

        databaseTester = new DataSourceDatabaseTester(dataSource);
        databaseTester.setSetUpOperation(DatabaseOperation.CLEAN_INSERT);
        databaseTester.setTearDownOperation(DatabaseOperation.NONE);
        databaseTester.setDataSet(getDataSet());
    }

    @Before
    public void insertDbUnitData() throws Exception {
        long time = System.currentTimeMillis();

        databaseTester.onSetup();

        long elapsed = System.currentTimeMillis() - time;
        System.out.println(getClass() + " Insert DB Unit Data took: " + elapsed);
    }

    @After
    public void cleanDbUnitData() throws Exception {
        databaseTester.onTearDown();
    }

    public IDataSet getDataSet() throws IOException, DataSetException {
        Set<String> filenames = getDataSets().getFilenames();

        IDataSet[] dataSets = new IDataSet[filenames.size()];
        Iterator<String> iterator = filenames.iterator();
        for(int i = 0; iterator.hasNext(); i++) {
            dataSets[i] = new FlatXmlDataSet(
                new FlatXmlProducer(
                    new InputSource(TEST_RESOURCES + iterator.next()), false, true
                )
            );
        }

        return new CompositeDataSet(dataSets);
    }

    public void setupHsqlDb() throws SQLException {
        Connection sqlConnection = DataSourceUtils.getConnection(dataSource);
        String databaseName = sqlConnection.getMetaData().getDatabaseProductName();
        sqlConnection.close();

        if("HSQL Database Engine".equals(databaseName)) {
            jdbcTemplate.update("SET DATABASE REFERENTIAL INTEGRITY FALSE;");

            // MD5
            jdbcTemplate.update("DROP FUNCTION MD5 IF EXISTS;");
            jdbcTemplate.update(
                "CREATE FUNCTION MD5(VARCHAR(226)) " +
                    "RETURNS VARCHAR(226) " +
                    "LANGUAGE JAVA " +
                    "DETERMINISTIC " +
                    "NO SQL " +
                    "EXTERNAL NAME 'CLASSPATH:org.apache.commons.codec.digest.DigestUtils.md5Hex';"
            );
        } else {
            jdbcTemplate.update("SET foreign_key_checks = 0;");
        }
    }

    protected abstract DataSet getDataSets();

    protected void flush() {
        sessionFactory.getCurrentSession().flush();
    }

    protected void clear() {
        sessionFactory.getCurrentSession().clear();
    }

    protected void setCurrentUser(User user) {
        if(user != null) {
            Authentication authentication = new UsernamePasswordAuthenticationToken(user,
                user, user.getAuthorities());

            SecurityContextHolder.getContext().setAuthentication(authentication);
        }
    }

    protected void setNoCurrentUser() {
        SecurityContextHolder.getContext().setAuthentication(null);
    }

    protected User setCurrentUser(long userId) {
        User user = userRepository.find(userId);

        if(user.getId() != userId) {
            throw new IllegalArgumentException("There is no user with id: " + userId);
        }

        setCurrentUser(user);

        return user;
    }

    protected User getCurrentUser() {
        return (User) SecurityContextHolder.getContext().getAuthentication().getPrincipal();
    }

}

Here is the relevant beans on my application context:

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
    <property name="locations" value="classpath:applicationContext.properties"/>
</bean>

<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"
      destroy-method="close">
    <property name="driverClass" value="${database.driver}"/>
    <property name="jdbcUrl" value="${database.url}"/>
    <property name="user" value="${database.username}"/>
    <property name="password" value="${database.password}"/>
    <property name="initialPoolSize" value="10"/>
    <property name="minPoolSize" value="10"/>
    <property name="maxPoolSize" value="50"/>
    <property name="idleConnectionTestPeriod" value="100"/>
    <property name="acquireIncrement" value="2"/>
    <property name="maxStatements" value="0"/>
    <property name="maxIdleTime" value="1800"/>
    <property name="numHelperThreads" value="3"/>
    <property name="acquireRetryAttempts" value="2"/>
    <property name="acquireRetryDelay" value="1000"/>
    <property name="checkoutTimeout" value="5000"/>
</bean>

<bean id="sessionFactory"
      class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
    <property name="dataSource" ref="dataSource"/>
    <property name="mappingResources">
        <list>
            <value>...</value>
        </list>
    </property>
    <property name="namingStrategy">
        <bean class="org.hibernate.cfg.ImprovedNamingStrategy"/>
    </property>
    <property name="hibernateProperties">
        <props>
            <prop key="javax.persistence.validation.mode">none</prop>

            <prop key="hibernate.dialect">${hibernate.dialect}</prop>
            <prop key="hibernate.hbm2ddl.auto">${hibernate.hbm2ddl.auto}
            </prop>
            <prop key="hibernate.generate_statistics">false</prop>

            <prop key="hibernate.show_sql">false</prop>
            <prop key="hibernate.format_sql">true</prop>

            <prop key="hibernate.cache.use_second_level_cache">false</prop>
            <prop key="hibernate.cache.provider_class">

            </prop>
        </props>
    </property>
</bean>

<bean class="org.springframework.orm.hibernate4.HibernateExceptionTranslator"/>

<bean id="transactionManager"
      class="org.springframework.orm.hibernate4.HibernateTransactionManager">
    <property name="sessionFactory" ref="sessionFactory"/>
</bean>

In order to try and insert less data, I allow each test class to pick a DataSet enum that only loads the data it needs. It's specified like this:

public enum DataSet {
    NONE(create()),
    CORE(create("core.xml")),
    USERS(combine(create("users.xml"), CORE)),
    TAGS(combine(create("tags.xml"), USERS)),

Could this be causing it to run slower rather than faster? The idea is that if I only want the core xml (languages, provinces, etc.), I only have to load those records. I thought this would make the test suite faster, but it's still too slow.

I can save some time by creating a separate xml dataset specifically designed for each test class. This chops out some of the insert statements. But even when I have 20 insert statements in a single xml dataset (thus, the minimum I/O loss other than in-lining the dataset right into java code directly), each test still takes .1 to .15 seconds during the initialization of the database data! I am in disbelief that it takes .15 seconds to insert 20 records into memory.

In my other project using Spring 3.0 and Hibernate 3.x, it takes 30 milliseconds to insert everything before each test, but it's actually inserting 100 or more rows per test. For the tests that only have 20 inserts, they are flying as if there was no delay at all. This is what I expected. I'm starting to think the problem is with Spring's annotations - or the way I have them setup in my DatabaseTest class. This is basically the only thing different now.

Also, my repositories are using the sessionFactory.getCurrentSession() instead of the HibernateTemplate. This is the first time I started using the annotation-based unit test stuff from Spring, since the Spring test classes are deprecated. Could that be the reason they are going slow?

If there's anything else you need to know to help figure it out, please let me know. I am sort of stumped.

EDIT: I put in the answer. The problem was hsqldb 2.2.x. Reverting to 2.0.0 fixes the problem.

Answer 1

That looks quite fast, IMHO. I've seen much slower integration tests. That said, there are various approaches that could make your tests faster:

• reduce the amount of inserted data
• avoid inserting the same data as the previous test if the data sets are identical and the previous test is a read-only test (which is often the case, especially if you rollback at the end of each test).

I guess doing it with DbUnit should be possible. If you're ready to use another framework, you could use my own DbSetup , which supports that out of the box.

Answer 2

The problem was Hsqldb 2.2.8. I reverted back to 2.0.0, and I got a 8-10x boost to performance or better instantly. Instead of taking 150-280 milliseconds, it went down to 7-15 (and sometimes 20) milliseconds.

My entire test suite (490 tests) now runs in just 18 seconds rather than 80.

I guess a note to everyone: avoid hsqldb 2.2.x. I think they added multithreading support which caused a performance problem with this type of use case.