TestNG retrying failed tests doesn't output the correct test results

Question

Setup: I have a class that extends the IRetryAnalyzer and have implemented a simple retry logic overriding the following method: public boolean retry(IT

Answer 1

Please consider the following test results with max. 2 Retries:

1. Passed => Overall ok!
2. Failed, Passed => Overall ok!
3. Failed, Failed, Passed => Overall ok!
4. Failed, Failed, Failed => Overall Fail!

What i did is to create a TestNg listener which extends the default behaviour and does some magic after all tests are finished.

public class FixRetryListener extends TestListenerAdapter {

    @Override
    public void onFinish(ITestContext testContext) {
        super.onFinish(testContext);

        // List of test results which we will delete later
        List testsToBeRemoved = new ArrayList<>();

        // collect all id's from passed test
        Set  passedTestIds = new HashSet<>();
        for (ITestResult passedTest : testContext.getPassedTests().getAllResults()) {
            passedTestIds.add(TestUtil.getId(passedTest));
        }

        Set  failedTestIds = new HashSet<>();
        for (ITestResult failedTest : testContext.getFailedTests().getAllResults()) {

            // id = class + method + dataprovider
            int failedTestId = TestUtil.getId(failedTest);

            // if we saw this test as a failed test before we mark as to be deleted
            // or delete this failed test if there is at least one passed version
            if (failedTestIds.contains(failedTestId) || passedTestIds.contains(failedTestId)) {
                testsToBeRemoved.add(failedTest);
            } else {
                failedTestIds.add(failedTestId);
            }
        }

        // finally delete all tests that are marked
        for (Iterator iterator = testContext.getFailedTests().getAllResults().iterator(); iterator.hasNext(); ) {
            ITestResult testResult = iterator.next();
            if (testsToBeRemoved.contains(testResult)) {
                iterator.remove();
            }
        }

    }

}

Basically i do 2 things:

1. Collect all passed test. If i encounter a failed test with at least one passed test i remove the failed test (That would cover case 2 and 3 from above)
2. Iterate over all failed test. If i encounter a failed test which previously failed i remove the current failed result. (That would cover case 3 and 4 actually). That also means i will only keep the first failed result if there are several failed results.

To identify a testresult i use the following simple hash function:

public class TestUtil {

    public static int getId(ITestResult result) {
        int id = result.getTestClass().getName().hashCode();
        id = 31 * id + result.getMethod().getMethodName().hashCode();
        id = 31 * id + (result.getParameters() != null ? Arrays.hashCode(result.getParameters()) : 0);
        return id;
    }
}

This approach does work with dataproviders as well but has one small limitation! If you use dataproviders with random values you'll run into problems:

@DataProvider(name = "dataprovider")
public Object[][] getData() {
    return new Object[][]{{System.currentTimeMillis()}};
}

For failed tests the dataprovider is reevaluated. Therefore you will get a new timestamp and the result1 and result2 for the same mehod will result in different hash values. Solution would be to include the parameterIndex in the getId() Method instead of parameters but it seems such a value is not included in ITestResult.

See this simple example as proof of concept:

@Listeners(value = FixRetryListener.class)
public class SimpleTest {

    private int count = 0;

    @DataProvider(name = "dataprovider")
    public Object[][] getData() {
        return new Object[][]{{"Run1"},{"Run2"}};
    }

    @Test(retryAnalyzer = RetryAnalyzer.class, dataProvider = "dataprovider")
    public void teste(String testName) {
        count++;
        System.out.println("---------------------------------------");
        System.out.println(testName + " " + count);
        if (count % 3 != 0) {
            Assert.fail();
        }

        count = 0;
    }

}

Will yield in:

Total tests run: 2, Failures: 0, Skips: 0