Rerun flaky JUnit test in case they failed

Question

I have a job A in Jenkins for my automated testing that is triggered if another job B build is successful. The job A run several tests. Some of the test are flaky so I would lik

Answer 1

Don't know of any plugin to run just the flaky/failed tests again, only the whole build. It should be possible, I just have not found any (and don't have enough time on my hand to write one). Here's what we did on a large java project where the build was ant based:

The build itself was pretty simple (using xml as formatter inside the junit ant task):
ant clean compile test

The build also accepted a single class name as parameter (using batchtest include section inside the junit ant task):
ant -Dtest.class.pattern=SomeClassName test

At the end of the jenkins job, we used the "Execute shell" build step. The idea was to search for all test results that had errors or failures, figure out the name of the class, then run that particular test class again. The file containing the failure will be overwritten, and the test collector at the end of the build will not see the flaky test failure, during the post build steps.

#!/bin/bash +x
cd ${WORKSPACE}
for i in $(seq 1 3); do 
    echo "Running failed tests $i time(s)"
    for file in `find -path '*/TEST-*.xml' | xargs grep 'errors\|failures' | grep '\(errors\|failures\)="[1-9]' | cut -d ':' -f 1`; do 
        class=`basename ${file} .xml | rev | cut -d '.' -f 1 | rev`
        ant -Dtest.class.pattern=${class} test
    done
done

After getting the build back under control, you definitely need to address the flaky tests. Don't let the green build fool you, there's still work to be done.