How would I produce JUnit test report for groovy tests, suitable for consumption by Jenkins/Hudson?

Question

I've written several XMLUnit tests (that fit in to the JUnit framework) in groovy and can execute them easily on the command line as per the groovy doco but I don't quite understand what else I've got to do for it to produce the xml output that is needed by Jenkins/Hudson (or other) to display the pass/fail results (like this) and detailed report of the errors etc (like this). (apologies to image owners)

Currently, my kickoff script is this:

def allSuite = new TestSuite('The XSL Tests')

//looking in package xsltests.rail.*
allSuite.addTest(AllTestSuite.suite("xsltests/rail", "*Tests.groovy")) 

junit.textui.TestRunner.run(allSuite)

and this produces something like this:

Running all XSL Tests...
....
Time: 4.141

OK (4 tests)

How can I make this create a JUnit test report xml file suitable to be read by Jenkins/Hudson?

Do I need to kick off the tests with a different JUnit runner?

I have seen this answer but would like to avoid having to write my own test report output.

Answer 1

After a little hackage I have taken Eric Wendelin's suggestion and gone with Gradle.

To do this I have moved my groovy unit tests into the requisite directory structure src/test/groovy/, with the supporting resources (input and expected output XML files) going into the /src/test/resources/ directory.

All required libraries have been configured in the build.gradle file, as described (in its entirety) here:

apply plugin: 'groovy'

repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.+'

    groovy module('org.codehaus.groovy:groovy:1.8.2') {
        dependency('asm:asm:3.3.1')
        dependency('antlr:antlr:2.7.7')
        dependency('xmlunit:xmlunit:1.3')
        dependency('xalan:serializer:2.7.1')
        dependency('xalan:xalan:2.7.1')
        dependency('org.bluestemsoftware.open.maven.tparty:xerces-impl:2.9.0')
        dependency('xml-apis:xml-apis:2.0.2')
    }
}

test {
    jvmArgs '-Xms64m', '-Xmx512m', '-XX:MaxPermSize=128m'

    testLogging.showStandardStreams = true //not sure about this one, was in official user guide

    outputs.upToDateWhen { false } //makes it run every time even when Gradle thinks it is "Up-To-Date"
}

This applies the Groovy plugin, sets up to use maven to grab the specified dependencies and then adds some extra values to the built-in "test" task.

One extra thing in there is the last line which makes Gradle run all of my tests every time and not just the ones it thinks are new/changed, this makes Jenkins play nicely.

I also created a gradle.properties file to get through the corporate proxy/firewall etc:

systemProp.http.proxyHost=10.xxx.xxx.xxx
systemProp.http.proxyPort=8080
systemProp.http.proxyUser=username
systemProp.http.proxyPassword=passwd

With this, I've created a 'free-style' project in Jenkins that polls our Mercurial repo periodically and whenever anyone commits an updated XSL to the repo all the tests will be run.

One of my original goals was being able to produce the standard Jenkins/Hudson pass/fail graphics and the JUnit reports, which is a success: Pass/Fail with JUnit Reports.

I hope this helps someone else with similar requirements.

Answer 2

I find the fastest way to bootstrap this stuff is with Gradle:

# build.gradle
apply plugin: 'groovy'

task initProjectStructure () << {
    project.sourceSets.all*.allSource.sourceTrees.srcDirs.flatten().each { dir ->
        dir.mkdirs()
    }
}

Then run gradle initProjectStructure and move your source into src/main/groovy and tests to test/main/groovy.

It seems like a lot (really it's <5 minutes of work), but you get lots of stuff for free. Now you can run gradle test and it'll run your tests and produce JUnit XML you can use in build/test-reports in your project directory.

Answer 3

Since you're asking for the purposes of exposing the report to Jenkins/Hudson, I'm assuming you have a Maven/Ant/etc build that you're able to run. If that's true, the solution is simple.

First of all, there's practically no difference between Groovy and Java JUnit tests. So, all you need to do is add the Ant/Maven junit task/plugin to your build and have it execute your Groovy junit tests (just as you'd do if they were written in Java). That execution will create test reports. From there, you can simply configure your Hudson/Jenkins build to look at the directory where the test reports get created during the build process.

Answer 4

You can write your own custom RunListener (or SuiteRunListener). It still requires you to write some code, but it's much cleaner than the script you've provided a link to. If you'd like, I can send you the code for a JUnit reporter I've written in JavaScript for Jasmine and you can 'translate' it into Groovy.