Is there hard evidence of the ROI of unit testing?

Question

Unit testing sounds great to me, but I\'m not sure I should spend any time really learning it unless I can convince others that is has significant value. I have to convince

Answer 1

More about TDD than strictly unit testing, here is a link to the Realizing quality improvement through test driven development: results and experiences of four industrial teams paper, by Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie Williams. paper published by Microsoft Empirical Software Engineering and Measurement (ESM) group and already mentionned here.

The team found was that the TDD teams produced code that is between 60% and 90% percent better (in terms of defect density) than non-TDD teams. However TDD teams took between 15% and 35% longer to complete their projects.

Answer 2

I do have one set of data points for this - from an experience that sold me on unit tests.

Many moons ago I was a fresh graduate working on a large VB6 project and had occasion to write a large body of stored procedure code. Of the subsystem I was writing it made up about 1/4 of the whole code base - around 13,000 LOC out of 50K or so.

I wrote a set of unit tests for the stored procedures but unit testing VB6 UI code is not really feasible without tools like Rational Robot; at least it wasn't back then.

The statistics from QA on the piece were that about 40 or 50 defects were raised on the whole subsystem, of which two originated from the stored procedures. That's one defect per 6,500 lines of code vs. 1 per 1,000-1,200 or so across the whole piece. Bear in mind also, that about 2/3 of the VB6 code was boilerplate code for error handling and logging, identical across all of the procedures.

Without too much handwaving you can ascribe at least an order-of-magnitude improvement in defect rates to the unit testing.

Answer 3

Yes. This is a link to a study by Boby George and Laurie Williams at NCST and a another by Nagappan et al. I'm sure there are more. Dr. Williams publications on testing may provide a good starting point for finding them.

[EDIT] The two papers above specifically reference TDD and show 15-35% increase in initial development time after adopting TDD, but a 40-90% decrease in pre-release defects. If you can't get at the full text versions, I suggest using Google Scholar to see if you can find a publicly available version.

Answer 4

Here's a great and entertaining read of a guy changing his company from within. It's not limited to TDD. http://jamesshore.com/Change-Diary/ Note that he didn't persuade the "bean counters" for quite some time and did "guerilla tactics" instead.

Answer 5

If you are also interested in evidence against unit testing here is one well researched and thought out article:

Why Most Unit Testing is Waste By James O Coplien (lean and agile guru)

Answer 6

Well, there are some large companies that require you to use unit testing but if you are a small company why mimic large ones?

For me when I started with unit testing , many years ago,(today we mostly use behavior model) it was because I could not control all the path in one application.

I was used to bottom first programming and a REPL so when I got Unit Test (One Test for Every Function) it was like bringing back a REPL to languages that where very much compile. It brought the fun back to every line of code I wrote. I felt god. I liked it. I didn't need a report to tell me that I begun writing better code faster. My boss didn't need a report to notice that because we where doing crazy stuff we suddenly never missed a deadline. My boss didn't need a report to notice that the number of "plain" bugs drop from (to many) to nearly nil because of this very strange thing of writing non-productive code.

As another poster already wrote, you don't use TDD to Test (verify). You write it to capture the specification, the behaviour of what your unit(object, module, function, class, server, cluster) works.

There are lot of failures and success stories of switching to a different model of developing software in a lot of companies.

I just started to use it whenever I had something new to write. There is a old saying that goes somewhat hard for me to translate to english but:

Start with something so simple that you don't notice that you do it. When training for a marathon, start by walking 9 meters and run 1 meter, repeat.