How do you tell that your unit tests are correct?

Question

I\'ve only done minor unit testing at various points in my career. Whenever I start diving into it again, it always troubles me how to prove that my tests are correct. How

Answer 1

Well, Dijkstra famously said:

"Testing shows the presence, not the absence of bugs"

IOW, how would you write a unit test for the function add(int, int)?

IOW, it's a tough one.

Answer 2

I had this same question, and having read the comments, here's what I now think (with due credit to the previous answers):

I think the problem may be that we both took the ostensible purpose of unit tests -- to prove the code is correct -- and applied that purpose to the tests themselves. That's fine as far as it goes, except the purpose of unit tests is not to prove that the code is correct.

As with all nontrivial endeavors, you can never be 100% sure. The correct purpose of unit tests is to reduce bugs, not eliminate them. Most specifically, as others have noted, when you make changes later on that might accidentally break something. Unit tests are just one tool to reduce bugs, and certainly should not be the only one. Ideally you combine unit testing with code review and solid QA in order to reduce bugs to a tolerable level.

Unit tests are much simpler than your code; it's not possible to make your code as simple as a unit test if your code does anything significant. If you write "small, granular" code that's easy to prove correct, then your code will consist of a huge number of small functions, and you're still going to have to determine whether they all work correctly in concert.

Since unit tests are inevitably simpler than the code they're testing, they're less likely to have bugs. Even if some of your unit tests are buggy, overall they're still going to improve the quality of your main codebase. (If your unit tests are so buggy that this isn't true, then likely your main codebase is a steaming pile as well, and you're completely screwed. I think we're all assuming a basic level of competence.)

If you DID want to apply a second level of unit testing to prove your unit tests correct, you could do so, but it's subject to diminishing returns. To look at it faux-numerically:

Assume that unit testing reduces the number of production bugs by 50%. You then write meta-unit tests (unit tests to find bugs in the unit tests). Say that this finds problems with your unit tests, reducing the production bug rate to 40%. But it took 80% as long to write the meta-unit tests as it did to write the unit tests. For 80% of the effort you only got another 20% of the gain. Maybe writing meta-meta-unit tests gives you another 5 percentage points, but now again that took 80% of the time it took to write the meta-unit tests, so for 64% of the effort of writing the unit tests (which gave you 50%) you got another 5%. Even with substantially more liberal numbers, it's not an efficient way to spend your time.

In this scenario it's clear that going past the point of writing unit tests isn't worth the effort.

Answer 3

For this to be a problem your code would have to be buggy in a way that coincidentally causes your tests to pass. This happened to me recently, where I was checking that a given condition (a) caused a method to fail. The test passed (i.e. the method failed), but it passed because another condition (b) caused a failure. Write your tests carefully, and make sure that unit tests test ONE thing.

Generally though, tests cannot be written to prove code is bug free. They're a step in the right direction.

Answer 4

This is one of the advantages of TDD: the code acts as a test for the tests.

It is possible that you'll make equivalent errors, but it is uncommon in my experience.

But I have certainly had the case where I write a test that should fail only to have it pass, which told me my test was wrong.

When I was first learning unit testing, and before I was doing TDD, I would also deliberately break the code after writing the test to ensure that it failed as I expected. When I didn't I knew the test was broken.

I really like Bob Martin's description of this as being equivalent to double entry bookkeeping.