How distinguish from which test suite was junit test run in case the same tests are run more times?

Question

I have JUnit main test suite. This suite contains many suites - for each testing configuration

  @RunWith(ProgressSuite.class)
    @SuiteClasses({

        SimpleTest.class, 
        AboutTest.class, 

        CDH4_JDBC_TestSuite.class, 
        CDH5_JDBC_TestSuite.class,
        CDH4_Metastore_TestSuite.class,
        CDH5_Metastore_TestSuite.class,
        CDH4_JDBC_Kerberos_TestSuite.class,
        CDH5_JDBC_Kerberos_TestSuite.class,
        CDH4_Metastore_Kerberos_TestSuite.class,
        CDH5_Metastore_Kerberos_TestSuite.class,


    })
    public class TestSuite {

    }

Suites for each testing configuration contains the same test cases, but contains different setUpClass() and tearDownClass() methods

@RunWith(Suite.class)
@SuiteClasses({

PerspectiveSwitchTest.class,

NewFolderFromToolbarTest.class,
RenameFolderFromToolbarTest.class,
RenameFileFromToolbarTest.class,
OpenFilePropertiesFromToolbarTest.class,
OpenFolderPropertiesFromToolbarTest.class,
DeleteFileFromToolbarTest.class,
DeleteFolderFromToolbarTest.class,
CopyPasteFolderFromToolbarTest.class,
CopyPasteFileFromToolbarTest.class,
CutPasteFolderFromToolbarTest.class,
CutPasteFileFromToolbarTest.class,

})

public class CDH4_JDBC_Kerberos_TestSuite {

    private static SWTWorkbenchBot bot = new SWTWorkbenchBot();
    private static AddNewEcosystemNavigator addNewEcosystemNavigator;
    private static EcosystemConfigurationLoader ecosystemConfigurationLoader;
    private static EcosystemConfiguration ecosystemConfiguration;
    private static GenericNavigator genericNavigator;

    @BeforeClass
    public static void setUpClass() {

        bot = new SWTWorkbenchBot();
        addNewEcosystemNavigator = new AddNewEcosystemNavigator();
        ecosystemConfigurationLoader = new EcosystemConfigurationLoader();
        genericNavigator = new GenericNavigator();

        ecosystemConfiguration = ecosystemConfigurationLoader
                .getDefaultCDH4JDBCKerberosEcosystemConfiguration();
        addNewEcosystemNavigator.addNewEcosystemManually(bot,
                ecosystemConfiguration);

    }

    @AfterClass
    public static void tearDownClass() {

        genericNavigator.closeDialogWindow();
        addNewEcosystemNavigator.discardEcosystem(bot, ecosystemConfiguration);

    }

}

I am using Jenkins and Tycho for building tests. When I run test suite and some tests fails, I am not able to distinguish on which configuration tests failed. In Jekins I can see only information eg NewFolderFromToolbarTest was runned 8 times (3 times failed, 5 times passed). Of course I am able get required information from log, but it is time consuming.

Is there any way how to get required information? eg Use different test structure, use different jenkins plugin, renamed method dynamically if even possible etc? any ideas please? thanks a lot

Answer 1

You could make the test classes abstract and have each configuration be a subclass

public class CDH4NewFolderFromToolbarTest extends AbstractNewFolderFromToolbarTest{
  //...

}

Then in your suite call the specific test classes

RunWith(Suite.class)
@SuiteClasses({

CDH4PerspectiveSwitchTest.class,

CDH4NewFolderFromToolbarTest.class,
CDH4RenameFolderFromToolbarTest.class,
CDH4RenameFileFromToolbarTest.class,
//...etc
})

public class CDH4_JDBC_Kerberos_TestSuite {

   //same as before

}

I would advocate that instead of reconfiguring in each subclass since the @BeforeClass and @AfterClass will only be called once if it is put in the suite.