Retry test case using TestNG while using ITestContext with test method

Question

I am using dataprovider method and a test method (with ITestContext parameter in the test method). A simplified example is as follows :

@DataProvider(name="Dataprovider")
public Object[][] dataprovider(){

    return new Object[][]{{1},{2,},{3}};
}

@Test(dataProvider="Dataprovider")
public void test(int data, ITestContext itx){

    System.out.println(data);
    org.testng.Assert.assertEquals(data, 3);
}

My Retry class and RetryListener classes are below :

public class RetryListener implements IAnnotationTransformer {

@Override
public void transform(ITestAnnotation testannotation, Class testClass,
        Constructor testConstructor, Method testMethod) {

    IRetryAnalyzer retry = testannotation.getRetryAnalyzer();

    if (retry == null)  {
        testannotation.setRetryAnalyzer(Retry.class);
    }
}
}

public class Retry implements IRetryAnalyzer {

private static int retryCount = 0;
private int maxRetryCount = 1;

// Below method returns 'true' if the test method has to be retried else 'false' 
//and it takes the 'Result' as parameter of the test method that just ran
    public boolean retry(ITestResult result) {
        if (retryCount < maxRetryCount) {
            System.out.println("Retrying test " + result.getName() + " with status "
                    + getResultStatusName(result.getStatus()) + " for the " + (retryCount+1) + " time(s).");
            retryCount++;
            return true;
        }
        retryCount = 0;
        return false;
    }

    public String getResultStatusName(int status) {
        String resultName = null;
        if(status==1)
            resultName = "SUCCESS";
        if(status==2)
            resultName = "FAILURE";
        if(status==3)
            resultName = "SKIP";
        return resultName;
    }

}

Expected : When test fails, the retry is called by TestNG, then the dataprovider should return the same values to the test method.

Observed : Dataprovider returns the same value but test method doesn't run and the retry terminates and next test starts (new values will now be returned by dataprovider)

But my retry does not enter the test method ( It is not expecting the (int data, ITestContext itx) for test method). If I remove ITestContext, the retry works.

ITestContext is a must for maintaining the test case context. So how to perform retry along with keeping the ITestContext in the test method.

Answer 1

Within a @Test method that's powered by a data provider, I think its difficult to put the ITestContext as an explicit parameter and expect it to be injected by TestNG, because TestNG wouldn't know how to inject it (especially at which position).

Please see if the below would work for you (If you notice, I am now explicitly also passing in the ITestContext via the data provider )

import org.testng.IRetryAnalyzer;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class RetryWithDataProvider {
    @DataProvider(name = "Dataprovider")
    public Object[][] dataprovider(ITestContext ctx) {

        return new Object[][]{{1, ctx}, {2, ctx}, {3, ctx}};
    }

    @Test(dataProvider = "Dataprovider", retryAnalyzer = Retry.class)
    public void test(int data, ITestContext itx) {
        System.out.println(data);
        System.out.println("Current Test name is : " + itx.getName());
        org.testng.Assert.assertEquals(data, 3);
    }

    public static class Retry implements IRetryAnalyzer {

        private static int retryCount = 0;
        private int maxRetryCount = 1;

        public boolean retry(ITestResult result) {
            if (retryCount < maxRetryCount) {
                System.out.println("Retrying test " + result.getName() + " with status "
                        + getResultStatusName(result.getStatus()) + " for the " + (retryCount + 1) + " time(s).");
                retryCount++;
                return true;
            }
            retryCount = 0;
            return false;
        }

        String getResultStatusName(int status) {
            String resultName = null;
            if (status == 1)
                resultName = "SUCCESS";
            if (status == 2)
                resultName = "FAILURE";
            if (status == 3)
                resultName = "SKIP";
            return resultName;
        }
    }
}

Here's my console output

[TestNG] Running:
  /Users/krmahadevan/Library/Caches/IntelliJIdea2016.3/temp-testng-customsuite.xml
1
Retrying test test with status FAILURE for the 1 time(s).

Test ignored.
1

java.lang.AssertionError: expected [3] but found [1]
Expected :3
Actual   :1
 <Click to see difference>


    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:513)
    at org.testng.Assert.assertEqualsImpl(Assert.java:135)
    at org.testng.Assert.assertEquals(Assert.java:116)
    at org.testng.Assert.assertEquals(Assert.java:389)
    at org.testng.Assert.assertEquals(Assert.java:399)
    at com.rationaleemotions.stackoverflow.RetryWithDataProvider.test(RetryWithDataProvider.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
    at org.testng.internal.Invoker.retryFailed(Invoker.java:998)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1200)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
    at org.testng.TestRunner.privateRun(TestRunner.java:756)
    at org.testng.TestRunner.run(TestRunner.java:610)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
    at org.testng.SuiteRunner.run(SuiteRunner.java:289)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
    at org.testng.TestNG.runSuites(TestNG.java:1133)
    at org.testng.TestNG.run(TestNG.java:1104)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:127)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

2
Retrying test test with status FAILURE for the 1 time(s).

Test ignored.
2

java.lang.AssertionError: expected [3] but found [2]
Expected :3
Actual   :2
 <Click to see difference>


    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:513)
    at org.testng.Assert.assertEqualsImpl(Assert.java:135)
    at org.testng.Assert.assertEquals(Assert.java:116)
    at org.testng.Assert.assertEquals(Assert.java:389)
    at org.testng.Assert.assertEquals(Assert.java:399)
    at com.rationaleemotions.stackoverflow.RetryWithDataProvider.test(RetryWithDataProvider.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
    at org.testng.internal.Invoker.retryFailed(Invoker.java:998)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1200)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
    at org.testng.TestRunner.privateRun(TestRunner.java:756)
    at org.testng.TestRunner.run(TestRunner.java:610)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
    at org.testng.SuiteRunner.run(SuiteRunner.java:289)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
    at org.testng.TestNG.runSuites(TestNG.java:1133)
    at org.testng.TestNG.run(TestNG.java:1104)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:127)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

3

===============================================
Default Suite
Total tests run: 5, Failures: 2, Skips: 2
===============================================