Unit test execution speed (how many tests per second?)

Question

What kind of execution rate do you aim for with your unit tests (# test per second)? How long is too long for an individual unit test?

I\'d be interested in knowing if

Answer 1

I judge my unit tests on a per test basis, not by by # of tests per second. The rate I aim for is 500ms or less. If it is above that, I will look into the test to find out why it is taking so long.

When I think a test is to slow, it usually means that it is doing too much. Therefore, just refactoring the test by splitting it up into more tests usually does the trick. The other times that I have noticed my tests running slow is when the test shows a bottleneck in my code, then a refactoring of the code is in order.

Answer 2

One of the most important rules about unit tests is they should run fast.

How long is too long for an individual unit test?

Developers should be able to run the whole suite of unit tests in seconds, and definitely not in minutes and minutes. Developers should be able to quickly run them after changing the code in anyway. If it takes too long, they won't bother running them and you lose one of the main benefits of the tests.

What kind of execution rate do you aim for with your unit tests (# test per second)?

You should aim for each test to run in an order of milliseconds, anything over 1 second is probably testing too much.

We currently have about 800 tests that run in under 30 seconds, about 27 tests per second. This includes the time to launch the mobile emulator needed to run them. Most of them each take 0-5ms (if I remember correctly).

We have one or two that take about 3 seconds, which are probably candidates for checking, but the important thing is the whole test suite doesn't take so long that it puts off developers running it, and doesn't significantly slow down our continuous integration build.

We also have a configurable timeout limit set to 5 seconds -- anything taking longer will fail.

Answer 3

The goal is 100s of tests per second. The way you get there is by following Michael Feather's rules of unit tests.

An important point that came up in a past CITCON discussion is that if your tests aren't this fast it is quite likely that you aren't getting the design benefits of unit testing.

Answer 4

We're currently at 270 tests in around 3.something seconds. There are probably around 8 tests that perform file IO.

These are run automatically upon a successful build of our libraries on every engineers machine. We have more extensive (and time consuming) smoke-testing that is done by the build machine every night, or can be started manually on an engineers machine.

As you can see we haven't yet reached the problem of tests being too time consuming. 10 seconds for me is the point where it starts to become intrusive, when we start to approach that it'll be something we'll take a look at. We'll likely move the lower level libraries, which are more robust since they change infrequently and have few dependencies, into the nightly builds, or a configuration where they're only executed by the build machine.

If you find it's taking more than a few seconds to run a hundred or so tests you may need to examine what you are classifying as a unit test and whether it would be better treated as a smoke test.

your mileage will obviously be highly variable depending on your area of development.