Can my build stipulate that my code coverage never get worse?

Question

I am using hudson CI to manage a straight java web project, using ant to build.

I would like to mandate that the unit test coverage never be worse than the previous build, thereby making sure any new code is always tested, or at least the coverage is continually improving.

Is there a hudson plugin that works this way?

Edit: I am currently using Emma, but would be willing to switch to another coverage app.

Also, as a clarification, I've seen the thresholds in some Hudson plugins, but that's not exactly what I'm after. For example what I'd like is that if coverage for Build #12 was 46% overall, and someone checked in Build #13 with 45% coverage, the build would break.

The reason I want to do this, is that I have a codebase with low test coverage. We don't have time to go back and retroactively write unit tests, but I'd like to make sure that the coverage keeps getting better.

UPDATE: Dan pointed out an edge case with my plan that will definitely be a problem. I think I need to rethink whether this is even a good idea.

Answer 1

Yes. Which coverage tool are you using?

The Cobertura plugin for Hudson definitely supports this. On the project configuration screen you can specify thresholds.

Alternatively, you can make Ant fail the build (rather than Hudson), by using the cobertura-check task.

EDIT: I'm not sure you can do precisely what you are asking for. Even if you could, it could prove problematic. For example, assume you have an average coverage of 75% but for one class you have coverage of 80%. If you remove that 80% class and all of its tests, you reduce the overall coverage percentage even though none of the other code is any less tested than previously.

Answer 2

This is kind of a hack, but we use it for similar reasons with Findbugs and Checkstyle. You can set up an Ant task to do the following (this can be split out into multiple tasks, but I'm combining them for brevity):

1. Run tests with coverage
2. Parse the coverage results and get the coverage percentage
3. Read tmp/lastCoverage.txt from last build (see step #5a)
4. Compare the current coverage percentage with the percentage read from lastCoverage.txt
5. 1. If percentage DIDN'T decrease, write the new percentage over the contents of tmp/lastCoverage.txt
   2. If percentage DID decrease, keep the original file and echo "COVERAGE FAILURE" (with ant's echo task).

Note that steps 2 through 5 don't necessarily need to be done with native Ant tasks - you could use something like Ant's javac task to run a Java program to do this for you.

Then, configure Hudson:

• Under "Source code management", make sure "Use Update" is checked. This will allow your lastCoverage.txt file to be retained between builds. Note that this could be problematic if you really, really need things to be cleaned between builds.
• Use the Hudson Text Finder plugin with a regular expression to search for "COVERAGE FAILURE" in the build output (make sure that "Also search console output" is checked for the plugin). The text finder plugin can mark the build unstable.

You can obviously replace things like the file name/path and console output to whatever fits within the context of your build.

As I mentioned above, this is rather hacky, but it's probably one of the few (only?) ways to get Hudson to compare things in the previous build to the current build.

Answer 3

Another approach would be to use the Sonar plugin for Hudson to maintain trending of coverage over time, and make it easier to assimilate and analyze results. It will also show coverage in context of other measures, such as checkstyle and pmd

Answer 4

Atlassian's Clover supports what you want. Have a look at the clover-check Ant task, specifically the historyDir attribute.