TestNG Automation Framework running tests individually, locally

Question

I am a beginner in TestNG. I am currently working in a company where they have a homegrown api automation framework. I am used to running my test case in isolation when local and using Cucumber. However, things are very tightly coupled in this framework. Let me give a briefing on different aspects of the framework:

Test Data Management: Test data are read through properties file. All the test case data are listed in the properties using key values per api module. The way the keys are numbered are by testcase type and a counter attached to it. So for example:

example.properties

.....
.....
    getTestCase38=TC_ID_38
    getDescription38= Description_38
    getUserID38=<some value>
    getUserType38=<some value>
    expectedUserType38=<some value>

    getTestCase39=TC_ID_39
    getDescription39= Description_39
    getUserID39=<some value>
    getUserType39=<some value>
    expectedUserType39=<some value>

    postTestCase1=TC_ID_1
    postDescription1= Description_1
    postUserID1=<some value>
    postUserType1=<some value>
    expectedUserType1=<some value>
...
...

Test Data Flow: The data is read from the properties file. A property file maps to a TestNG class. The naming convention of these methods in a TestNG class are as follows:

public class ModuleA extends ModuleACommon{    

int getcount = 0;
int postcount = 0;

    @Test(priority=20)
    public void testMoudleAGetPositive38(){
     callMethodToReadData(getcount++);
     ....
    }   

    @Test(priority=21)
    public void testModuleAGetPositive39(){
     callMethodToReadData(getcount++);
     ....
     ....
    } 
    ......
    ......

    @Test(priority=30)
    public void testModuleAPostPositive1(){
     callMethodToReadData(postcount++);
     ....
     ....
    }   
   ......
   ......
}

Note that when callMethodToReadData is called the count is incremented and it is used to build the key to access the test data from the properties file. The test cases are set to a priority in sequential order, this ensures that the proper key is constructed. This is painful because if I want to add a new test case following this pattern. Lets say I want to add a test case and assign it a priority of 21 then I will have to adjust the priority for all the test methods after 21 including the 21st priority test case.

For short term: I am trying to find an easiest way to run one test in isolation. There are dirty ways of accomplishing this. For example, I can just comment out other test case and reassign the appropriate count values but this is too inefficient.

Long Term: I would love to hear opinions and suggestions to improve this in the long run. I have couple of ideas myself for long term improvements, however I was hoping to get more insightful suggestions from here.

Also, let me know if there are any hacks that could solve the problem of adding test cases and not having to reorganize the priorities (I have a solution that I have already implemented where I just use a unique key that is more descriptive of the actual test case).

Answer 1

Consolidating some of the problems that can be seen with this approach :

1. No support for parallel execution : The way your tests are built, if I were to attempt to run them in parallel, the entire system would collapse, because the method that fetches data from a data source (properties file in your case) is dependent on a non thread safe data member in the class.
2. As you already called out, cannot add new tests with ease nor have the flexibility to run them as standalone tests.
3. Re-ordering of tests

In order to be able to run a test independently is to be able to change your key. You key currently is relying on some int value, which needs to be passed off to the data source so that you can retrieve properties for it.

I would suggest that you change this to instead start being the method name of the @Test . Since you are saying that for every test class you would be having a properties file and since Java does not let you define two methods with the same name (of-course method overloading will let you do that, but since we are talking about @Test methods, we can conveniently ignore that part), your keys would still be unique if you had them depend on the method name. Now you don't need to be explicitly referring to the method name also. From within your @Test method, if you invoked callMethodToReadData() , then your callMethodToReadData() can easily retrieve the currently running @Test method's name by merely calling org.testng.Reporter.getCurrentTestResult().getMethod().getMethodName() (Remember that TestNG ensures that there's a valid ITestResult object when you query the thread local variable Reporter.getCurrentTestResult from within a @Test method)

This will now free you off from the dependency on a counter, to form your key. Your properties file is much more readable because the key now represents the method name, so its a lot more clear as to, to which test method does the data belong to.

You also don't have to alter priorities when adding a new test (of-course you would need to adjust priorities if you really want to change the order of the tests, because that is what priorities are for.. soft dependency )

You can get a bit more fancy by even building an implementation of org.testng.IAnnotationTransformer wherein you can enable/disable @Test methods at will based on a JVM argument which you can read. So that literally lets you run any @Test method at will. If you are using maven as your build tool, then you really dont need the transformer because maven surefire plugin by itself lets you do that.

In the long run, I believe you need to start looking at dataprovider and perhaps even factories. You might also want to get away from the notion of properties and instead move over to something sophisticated such as JSON/YAML/XML or just fall back to excel spreadsheets, because in excel spreadsheets you can literally start creating a full fledged RDBMS by visualising each sheet as a table and then having one table refer to the other using keys. You can then go about building a sophisticated excel data provider, which is capable of extracting data from the spreadsheet and populating a POJO (you would need to create one, which models your spreadsheet row of data). This is what we ended up building and open sourcing as part of building SeLion . To know more about the excel data provider that I am talking about, you can refer here