How do code coverage tools work?

Question

How do code coverage tools like NCover know what parts of the code were executed and what parts were not?

Answer 1

It requires that you run your tests once with code coverage analysis enables, and then simply counts the number of blocks (that is, scope blocks) covered and compares to the total number of blocks in the project(s) you are testing.

The basic reasoning is that if each possible combination of code blocks is covered, all code paths are covered¹. The main argument against putting too much weight in code coverage numbers is that "easy" blocks like getters and setters, that give no real value (and could hardly go wrong...) count just as much as more error-prone blocks of code.

---

^{1) As noted by Ira Baxter in a comment, the previous wording of this sentence was incorrect. Please read the comments for some discussion on this.}

Answer 2

Here's a technical paper on How to Implement test coverage tools for arbitrary languages.

My company builds a family of test coverage tools for Java, C#, C++, PHP, COBOL, PLSQL, ... based on this principle.

Answer 3

Quote straight from the NCover FAQ: NCover reports the percentage of branches in the code that have been taken throughout the course of your automated testing. It achieves this by instrumenting the source code at each branch, and writing the 'hit' points to a file. These 'hit' points are then compared to the total possible points that could have been 'hit'.

Answer 4

I know this is question is old but if you are still interested you can see an example of how such instrumentation is performed for .NET applications by looking at the open source project OpenCover.

OpenCover inserts instrumentation points at significant points in the code.

1. For code line coverage it uses the sequence points taken from a PDB file
2. For branch coverage it instruments COND_BRANCH instructions by instrumenting the jump target(s) and the next instruction after the branch instruction i.e. no jump.
3. For method instrumentation it instruments the first instruction of any method.

All of these rules are applied in CoverageInstrumentation.cpp after the appropriate points have been located using Mono.Cecil and passed to the profiler from the console host.

The source code to PartCover is also available (as indicated) but this is much harder to follow but it also uses sequence points from PDBs to determine where it instruments the code.

Answer 5

From this source:

NCover uses the .NET Framework profiler API to monitor an application's execution. When a method is loaded by the CLR, NCover retrieves the IL and replaces it with instrumented IL code

So in short, it hooks itself into the just-in-time compilation.

Not all tools work the same way though. Other tools work by modifying the bytecode of your application after the code has been compiled.