Jenkins - How to find out which tests failed repeatedly?

Question

I am new to CI & Jenkins. I have a Java project which runs Testng based automated tests. The tests run regularly as a job in Jenkins. Sometimes, the job fails repeatedly for

Answer 1

You can take the number of times a test failed from the build's object in a pipeline: you can make a small report and attach it to your build like so:

@NonCPS
def getRepeatedlyFailingTests(int timesFailedAtLeast = 2) {
  currentBuild.rawBuild.getAction(hudson.tasks.junit.TestResultAction.class)
    .getFailedTests()
    // Keep only tests that failed at least twice
    .findAll { it.age >= timesFailedAtLeast }
    .collect { [ "${it.className}.${it.name}".trim(), it.age ] }
}

def saveRepeatedlyFailingTestsReport() {
  def header = [ "test", "times-failed" ]
  def records = getRepeatedlyFailingTests()
  def report = "repeatedly-failing-tests.csv"
  writeCSV file: report, records: [ header ] + records, format: CSVFormat.EXCEL
  archiveArtifacts report
}

Call saveRepeatedlyFailingTestsReport after you gather your test results and you'll see the report in your latest build artifacts link.

Answer 2

You can use the Test Result Analyzer plugin from Jenkins to analyse the trends of tests.

Test Result Analyzer plugin

Video Tutorial from youtube

Installation:

1. In Jenkins Dashboard, click on Manage Jenkins
2. From the menu select Manage Plugins
3. Click on Available tab and input analyzer in the Filter search box
4. Click the check box to the left of Test Results Analyzer and click the Install without restart button

Usage:

1. Go to the project and click on Configure
2. Goto Post-build Actions section and select Publish JUnit test result report from the Add post-build action dropdown
3. In the Test report XMLs field provide the path of the TestNg/JUnit xml in the project. For example - **/target/test-results/*.xml in my case
4. Save the configuration
5. Then after a successful build, goto your project, and click on the Test Results Analyzer on the left side menu