What can be alternative metrics to code coverage?

Question

Code coverage is propably the most controversial code metric. Some say, you have to reach 80% code coverage, other say, it\'s superficial and does not say anything about your te

Answer 1

Probably not only measuring the code covered (touched) by the unit tests but how good the assertions are.

One metric easy to implement is to measure the size of the Assert.AreEqual

You can create your own Assert implementation calling Assert.AreEqual and measuring the size of the object passed as second parameter.

Answer 2

Crap4j is one fairly good metrics that I'm aware of...

Its a Java implementation of the Change Risk Analysis and Predictions software metric which combines cyclomatic complexity and code coverage from automated tests.

Answer 3

Using code coverage on it's own is mostly pointless, it gives you only insight if you are looking for unnecessary code.

Using it together with unit-tests and aiming for 100% coverage will tell you that all the 'tested' parts (assumed it was all successfully too) work as specified in the unit-test.

Writing unit-tests from a technical design/functional design, having 100% coverage and 100% successful tests will tell you that the program is working like described in the documentation.

Now the only thing you need is good documentation, especially the functional design, a programmer should not write that unless (s)he is an expert of that specific field.

Answer 4

Code Coverage is just an indicator and helps pointing out lines which are not executed at all in your tests, which is quite interesting. If you reach 80% code coverage or so, it starts making sense to look at the remaining 20% of lines to identify if you are missing some use case. If you see "aha, this line gets executed if I pass an empty vector" then you can actually write a test which passes an empty vector.

As an alternative I can think of, if you have a specs document with Use Cases and Functional Requirements, you should map the unit tests to them and see how many UC are covered by FR (of course it should be 100%) and how many FR are covered by UT (again, it should be 100%).

If you don't have specs, who cares? Anything that happens will be ok :)

Answer 5

Bug metrics are also important:

• Number of bugs coming in
• Number of bugs resolved

To detect for instance if bugs are not resolved as fast as new come in.

Answer 6

The value in code coverage is it gives you some idea of what has been exercised by tests. The phrase "code coverage" is often used to mean statement coverage, e.g., "how much of my code (in lines) has been executed", but in fact there are over a hundred varieties of "coverage". These other versions of coverage try to provide a more sophisticated view what it means to exercise code.

For example, condition coverage measures how many of the separate elements of conditional expressions have been exercised. This is different than statement coverage. MC/DC "modified condition/decision coverage" determines whether the elements of all conditional expressions have all been demonstrated to control the outcome of the conditional, and is required by the FAA for aircraft software. Path coverage meaures how many of the possible execution paths through your code have been exercised. This is a better measure than statement coverage, in that paths essentially represent different cases in the code. Which of these measures is best to use depends on how concerned you are about the effectiveness of your tests.

Wikipedia discusses many variations of test coverage reasonably well. http://en.wikipedia.org/wiki/Code_coverage