问题
Similar to the solution posted here TestNG retrying failed tests doesn't output the correct test results, I'm trying to remove a (duplicate) test result using a test listener during onFinish(ITestContext context).
Although the removal of the result with context.getFailedTests().removeResult(result) appears to work fine (the result actually is removed), there seems to be "some other spot" where the results are being pulled from cause the build still fails.
Also note that when I run the sample test from the article above (which has one duplicate failure to be removed and one passed test), I get a difference in "test results" (not cleaned up as expected) vs. "suite results" (duplicate failure removed as expected).
And, where is the reporting pulling the results from to decide whether to fail the build? Or is it just that it's pulling the results before I'm cleaning up the failed tests ... ?
===============================================
Default test
Tests run: 3, Failures: 2, Skips: 0
===============================================
===============================================
Default suite
Total tests run: 2, Failures: 1, Skips: 0
===============================================
EDIT: Just to clarify, we're running those tests with maven, and they are ITs, so we run them with the failsafe plugin. The problem is that even though it appears that the tests are removed, mvn verify still fails the build as it considers build failures to be found regardless.
And also if run such a test from Eclipse, even though the tests are removed, failures are still being printed in the log when the suite finishes.
About RetryAnalyzer: I would not consider using RetryAnalyzer good/best practice at all, but if you find yourself in a situation where you need to solve the problem, e.g. you inherited a test suite that is relying on RetryAnalyzer, you may find this useful.
回答1:
Try using this code:
ListenerApadter:
public class MyTestListenerAdapter extends TestListenerAdapter {
@Override
public void onTestFailure(ITestResult result) {
if (result.getMethod().getRetryAnalyzer() != null) {
MyRetryAnalyzer retryAnalyzer = (MyRetryAnalyzer)result.getMethod().getRetryAnalyzer();
if(retryAnalyzer.isRetryAvailable()) {
result.setStatus(ITestResult.SKIP);
} else {
result.setStatus(ITestResult.FAILURE);
}
Reporter.setCurrentTestResult(result);
}
}
@Overrride
public void onFinish(ITestContext context) {
Iterator<ITestResult> failedTestCases =context.getFailedTests().getAllResults().iterator();
while (failedTestCases.hasNext()) {
System.out.println("failedTestCases");
ITestResult failedTestCase = failedTestCases.next();
ITestNGMethod method = failedTestCase.getMethod();
if (context.getFailedTests().getResults(method).size() > 1) {
System.out.println("failed test case remove as dup:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
} else {
if (context.getPassedTests().getResults(method).size() > 0) {
System.out.println("failed test case remove as pass retry:" + failedTestCase.getTestClass().toString());
failedTestCases.remove();
}
}
}
}
}
RetryAnalizer:
public class MyRetryAnalyzer implements IRetryAnalyzer {
private static int MAX_RETRY_COUNT = 3;
AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);
public boolean isRetryAvailable() {
return (count.intValue() > 0);
}
@Override
public boolean retry(ITestResult result) {
boolean retry = false;
if (isRetryAvailable()) {
System.out.println("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT);
retry = true;
count.decrementAndGet();
}
return retry;
}
}
POM.xml -> Surefire Configuration:
This is where you should configure "overwrite" surefire listener wich has their own counters.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<suiteXmlFiles><suiteXmlFile>${basedir}/testng.xml</suiteXmlFile></suiteXmlFiles>
<properties>
<property>
<name>listener</name>
<value>Utils.MyTestListenerAdapter,Utils.MyRetryAnalizer</value>
</property>
</properties>
回答2:
I ended up going for a solution that uses a suite listener.
The solution may not fully clean up what TestNG / your tests log to the console, but if you use the TestNG Jenkins plugin, and had a failure first on every test, and then success, the test run ends up to be green, which I guess is the most important thing.
And yes, we run mvn integration-test (not mvn verify) and let the TestNG plugin deal with pass / fail. The solution is quite similar / builds on what was posted here.
import java.util.Map;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
/**
* {@link ISuiteListener} implementation to clean up duplicate test results caused by retrying tests using the
* {@link RetryAnalyzer}
*/
public class SuiteResultListener implements ISuiteListener {
private static final Logger LOG = LogManager.getLogger();
@Override
public void onStart(ISuite suite) {
}
@Override
public void onFinish(ISuite suite) {
LOG.info("Cleaning up duplicate test failures in suite '" + suite.getName() + "' ...");
final Map<String, ISuiteResult> results = suite.getResults();
int removedFailures = 0;
for (ISuiteResult result : results.values()) {
final ITestContext testContext = result.getTestContext();
removedFailures += TestListenerUtil.cleanUpDuplicateFailures(testContext);
}
LOG.info("Removed " + removedFailures + " duplicate test failure(s) from suite '" + suite.getName() + "'");
}
}
And here's the magic that happens in the TestListenerUtil class:
public static int cleanUpDuplicateFailures(ITestContext testContext) {
final String testContextName = testContext.getName();
int removedFailures = 0;
LOG.info("Cleaning up failures in test context '" + testContextName + "' ...");
final Set<ITestResult> failedTests = testContext.getFailedTests().getAllResults();
if (failedTests.isEmpty()) {
LOG.info("There are no failures in test context '" + testContextName + "'\n");
} else {
// collect all id's from passed test
final Set<Integer> passedTestIds = new HashSet<>();
final Set<ITestResult> passedTests = testContext.getPassedTests().getAllResults();
LOG.info("Analyzing " + passedTests.size() + " passed test(s)");
for (ITestResult result : passedTests) {
final int testId = TestListenerUtil.getId(result);
passedTestIds.add(testId);
LOG.info(" Passed test " + TestListenerUtil.getName(result) + ": #" + testId + " @ "
+ getStartTime(result));
}
// check which failed test results should be removed
final List<Integer> resultsToBeRemoved = new ArrayList<>();
final Set<Integer> failedTestIds = new HashSet<>();
LOG.info("Analyzing " + failedTests.size() + " failed test(s)");
for (ITestResult result : failedTests) {
final int testId = TestListenerUtil.getId(result);
final String name = TestListenerUtil.getName(result);
// if we saw this test pass or fail before we mark the result for deletion
if (failedTestIds.contains(testId) || passedTestIds.contains(testId)) {
LOG.info(" Adding test " + name + " to be removed: #" + testId + " @ " + getStartTime(result));
resultsToBeRemoved.add(testId);
} else {
LOG.info(" Remembering failed test " + name + ": #" + testId + " @ " + getStartTime(result));
failedTestIds.add(testId);
}
}
// finally delete all duplicate failures (if any)
final int duplicateFailures = resultsToBeRemoved.size();
if (duplicateFailures > 0) {
LOG.info("Cleaning up failed tests (expecting to remove " + resultsToBeRemoved.size()
+ " result(s)) ...");
for (ITestResult result : testContext.getFailedTests().getAllResults()) {
final int testId = TestListenerUtil.getId(result);
final String info = TestListenerUtil.getName(result) + ": #" + testId + " @ "
+ getStartTime(result);
if (resultsToBeRemoved.contains(testId)) {
LOG.info(" Removing failed test result " + info);
testContext.getFailedTests().removeResult(result);
resultsToBeRemoved.remove((Integer) testId);
removedFailures++;
} else {
LOG.info(" Not removing failed test result " + info);
}
}
}
if (removedFailures == duplicateFailures) {
LOG.info("Removed " + removedFailures + " failed test result(s) in '" + testContextName + "'\n");
} else {
LOG.warn("Removed " + removedFailures + " failed test result(s) in '" + testContextName
+ "' (expected to remove " + duplicateFailures + ")\n");
}
}
return removedFailures;
}
With those two additional utils methods:
public static String getName(ITestResult result) {
final List<String> parameters = new ArrayList<>();
if (result.getParameters() != null) {
for (Object parameter : result.getParameters()) {
if (parameter instanceof TestResult && ((TestResult) parameter).getStatus() < 0) {
// TestResult.toString() will explode with status < 0, can't use the toString() method
parameters.add(parameter.getClass().getName() + "@" + parameter.hashCode());
} else {
parameters.add(parameter == null ? "null" : parameter.toString());
}
}
}
return result.getTestClass().getRealClass().getSimpleName() + "." + result.getMethod().getMethodName() + "("
+ StringUtils.join(parameters, ",") + ")";
}
public static int getId(ITestResult result) {
final HashCodeBuilder builder = new HashCodeBuilder();
builder.append(result.getTestClass().getRealClass());
builder.append(result.getMethod().getMethodName());
builder.append(result.getParameters());
return builder.toHashCode();
}
And also, if you're interested how our RetryAnalyzer works, see below.
One thing that is important to understand, is that we're taking the the parameters of the test method into account in both the RetryAnalyzer and the duplicate result clean up. Those are relevant cause we often work with DataProviders.
import java.util.HashMap;
import java.util.Map;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryAnalyzer implements IRetryAnalyzer {
private static final Logger LOG = LogManager.getLogger();
private static Integer maxRetries;
private final Map<Integer, Integer> retryCount = new HashMap<>();
@Override
public boolean retry(ITestResult result) {
// only re-try failures
if (result.getStatus() == ITestResult.FAILURE) {
final String testName = TestListenerUtil.getName(result);
final int count = getRetryCount(result);
final int maxRetriesAllowed = getMaxRetriesAllowed();
if (count < maxRetriesAllowed) {
retryCount.put(TestListenerUtil.getId(result), count + 1);
LOG.info("Retrying test (attempt " + (count + 1) + "/" + maxRetriesAllowed + "): " + testName);
return true;
} else {
LOG.error("Failing test after " + count + " retries: " + testName);
}
}
return false;
}
public boolean canRetry(ITestResult result) {
return result.getStatus() == ITestResult.FAILURE && getRetryCount(result) < getMaxRetriesAllowed();
}
private int getRetryCount(ITestResult result) {
final int testId = TestListenerUtil.getId(result);
return retryCount.containsKey(testId) ? retryCount.get(testId) : 0;
}
public static int getMaxRetriesAllowed() {
return maxRetries == null ? Config.MAX_TEST_RETRIES : maxRetries;
}
}
来源：https://stackoverflow.com/questions/29104427/remove-duplicate-failed-testng-result-via-test-listener