How can I speed up my PHPUnit + DBUnit test suite execution?

Question

I'm running in to some real speed issues with PHPUnit/DBUnit. Anything that extends PHPUnit_Extensions_Database_TestCase takes forever to run. With 189 tests, the suite takes around 8-9 minutes. I was kind of hoping it would take 30 seconds at most ;-)

It looks like restoring the database to its initial state is the process that takes the time, so we've made our datasets as small as possible and limit the number of tables we require for each test case. I am using fixtures and sharing as much as possible.

Are there any settings or modifications I can use to speed the execution up? Looking at what the MySQL server is doing throughout the tests it seems that lots of truncate/inserting is happening, but surely it would be faster to pack the test data sets into temporary tables and then simply select into them for each test?

The driver I'm using is PDO/MySQL with an XML test dataset.

Answer 1

Upon Googling I've managed to reduce the time it takes from 10 minutes down to 1 minute. It turns out that changing some InnoDB configuration settings in my.ini/my.cnf will help.

Setting innodb_flush_log_at_trx_commit = 2 seems to do the job. After you change it, restart your MySQL server.

More on dev.mysql.com: innodb_flush_log_at_trx_commit

The setting controls how ACID compliant the flushing of the logs is. The default value is 1 which is full ACID compliance which means

the log buffer is written out to the log file at each transaction commit and the flush to disk operation is performed on the log file.

With a value of 2, the following happens:

The log buffer is written out to the file at each commit, but the flush to disk operation is not performed on it.

The key difference here is that because the log isn't written out at every commit, an operating system crash or power outage can wipe it out. For production, stick to a value of 1. For local development with a test database, the value of 2 should be safe.

If you're working with data that will be transferred to the live database, I would suggest sticking with the value of 1.

Answer 2

The fixture creation in DbUnit is extremely slow. It takes 1.5 sec every time with core2duo e8400 4gb kingston 1333. You can find the bottleneck with xdebug and fix it (if you can), or you can do one of the following:

1.)

You can run only test files you currently develop with a custom bootstrap xml:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="http://phpunit.de/phpunit.xsd"
         backupGlobals="false"
         verbose="true"
         bootstrap="test/bootstrap.php">
    <testsuites>
        <testsuite>
            <directory>test/integration</directory>
            <exclude>test/integration/database/RoleDataTest.php</exclude>
        </testsuite>
    </testsuites>
    <php>
        <env name="APPLICATION_MODE" value="test"/>
    </php>
</phpunit>

The exclude part is important here. You can use test groups too.

2.)

namespace test\integration;


abstract class AbstractTestCase extends \PHPUnit_Extensions_Database_TestCase
{
    static protected $pdo;
    static protected $connection;

    /**
     * @return \PHPUnit_Extensions_Database_DB_IDatabaseConnection
     */
    public function getConnection()
    {
        if (!isset(static::$pdo)) {
            static::$pdo = new \PDO('pgsql:host=localhost;port=5432;dbname=dobra_test', 'postgres', 'inflames', array(\PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION));
            static::$connection = $this->createDefaultDBConnection(static::$pdo);
        }
        return static::$connection;
    }

    /**
     * @return \PHPUnit_Extensions_Database_Operation_DatabaseOperation
     */

    static protected $fixtureSet = false;

    protected function getSetUpOperation()
    {
        $c = get_class($this;
        if (!$c::$fixtureSet) {
            $c::$fixtureSet = true;
            return \PHPUnit_Extensions_Database_Operation_Factory::CLEAN_INSERT(true);
        }
        return \PHPUnit_Extensions_Database_Operation_Factory::NONE();
    }

    static protected $dataSet;

    /**
     * @return \PHPUnit_Extensions_Database_DataSet_IDataSet
     */
    public function getDataSet()
    {
        $c = get_class($this;
        if (!isset($c::$dataSet)) {
            $c::$dataSet = $this->createDataSet();
        }
        return $c::$dataSet;
    }

    /**
     * @return \PHPUnit_Extensions_Database_DataSet_IDataSet
     */
    abstract protected function createDataSet();

    protected function dataSetToRows($tableName, array $ids)
    {
        $transformer = new DataSetRowsTransformer($this->getDataSet());
        $transformer->findRowsByIds($tableName, $ids);
        $transformer->cutColumnPrefix();
        return $transformer->getRows();
    }

}

You can override the TestCase. In this example you will use only one pdo connection by every test case (you can inject it to your code with dependency injection), by overriding setup operation you can set the fixture only once per testcase or only once for every test (depends on self:: or $cls = get_class($this); $cls:: ). (PHPUnit has bad design, it creates new instance by every test call, so you have to hack with the class names to store variables per instance or per class.) By this scenario you have to write the tests to depend on eachother with @depend annotation. For example you can delete the same row you created in the previous test.

By this test code 1.5 secs instead of 6 x 1.5 = 9 secs :

namespace test\integration\database;

use Authorization\PermissionData;
use test\integration\AbstractTestCase;
use test\integration\ArrayDataSet;

class PermissionDataTest extends AbstractTestCase
{
    static protected $fixtureSet = false;
    static protected $dataSet;

    /** @var PermissionData */
    protected $permissionData;

    /**
     * @return \PHPUnit_Extensions_Database_DataSet_IDataSet
     */
    public function createDataSet()
    {
        return new ArrayDataSet(array(
            'permission' => array(
                array('permission_id' => '1', 'permission_method' => 'GET', 'permission_resource' => '^/$'),
                array('permission_id' => '2', 'permission_method' => 'POST', 'permission_resource' => '^/$'),
                array('permission_id' => '3', 'permission_method' => 'DELETE', 'permission_resource' => '^/$')
            ),
            'user' => array(
                array('user_id' => '1', 'user_name' => 'Jánszky László', 'user_email' => 'a@b.d', 'user_salt' => '12435')
            ),
            'user_permission' => array(
                array('user_permission_id' => '1', 'user_id' => '1', 'permission_id' => '1'),
                array('user_permission_id' => '2', 'user_id' => '1', 'permission_id' => '2')
            ),
            'role' => array(
                array('role_id' => '1', 'role_name' => 'admin')
            ),
            'role_permission' => array(
                array('role_permission_id' => '1', 'role_id' => '1', 'permission_id' => '1')
            ),
            'permission_cache' => array(
                array('permission_cache_id' => '1', 'user_id' => '1', 'permission_id' => '1'),
                array('permission_cache_id' => '2', 'user_id' => '1', 'permission_id' => '2'),
            )
        ));
    }

    public function testReadAllShouldReturnEveryRow()
    {
        $this->assertEquals($this->permissionData->readAll(), $this->dataSetToRows('permission', array(3, 2, 1)));
    }

    /** @depends testReadAllShouldReturnEveryRow */

    public function testReadAllByRoleIdShouldReturnEveryRowRelatedToRoleId()
    {
        $this->assertEquals($this->permissionData->readAllByRoleId(1), $this->dataSetToRows('permission', array(1)));
    }

    /** @depends testReadAllByRoleIdShouldReturnEveryRowRelatedToRoleId */

    public function testReadAllByUserIdShouldReturnEveryRowRelatedToUserId()
    {
        $this->assertEquals($this->permissionData->readAllByUserId(1), $this->dataSetToRows('permission', array(2, 1)));
    }

    /** @depends testReadAllByUserIdShouldReturnEveryRowRelatedToUserId */

    public function testCreateShouldAddNewRow()
    {
        $method = 'PUT';
        $resource = '^/$';
        $createdRow = $this->permissionData->create($method, $resource);
        $this->assertTrue($createdRow['id'] > 0);
        $this->assertEquals($this->getDataSet()->getTable('permission')->getRowCount() + 1, $this->getConnection()->getRowCount('permission'));
        return $createdRow;
    }

    /** @depends testCreateShouldAddNewRow */

    public function testDeleteShouldRemoveRow(array $createdRow)
    {
        $this->permissionData->delete($createdRow['id']);
        $this->assertEquals($this->getDataSet()->getTable('permission')->getRowCount(), $this->getConnection()->getRowCount('permission'));
    }

    /** @depends testDeleteShouldRemoveRow */

    public function testDeleteShouldRemoveRowAndRelations()
    {
        $this->permissionData->delete(1);
        $this->assertEquals($this->getDataSet()->getTable('permission')->getRowCount() - 1, $this->getConnection()->getRowCount('permission'));
        $this->assertEquals($this->getDataSet()->getTable('user_permission')->getRowCount() - 1, $this->getConnection()->getRowCount('user_permission'));
        $this->assertEquals($this->getDataSet()->getTable('role_permission')->getRowCount() - 1, $this->getConnection()->getRowCount('role_permission'));
        $this->assertEquals($this->getDataSet()->getTable('permission_cache')->getRowCount() - 1, $this->getConnection()->getRowCount('permission_cache'));
    }

    public function setUp()
    {
        parent::setUp();
        $this->permissionData = new PermissionData($this->getConnection()->getConnection());
    }
}

3.)

Another solution to create the fixture only once per project, and after that use every test in transactions and rollback after every test. (This does not work if you have pgsql deferred code which needs commit to check the constraints.)