What are the downsides using random values in Unit Testing?

Question

I\'m talking about a large scales system, with many servers and non deterministic input in high capacity. When i say non deterministic i\'m talking about messages that are s

Answer 1

Downsides

Firstly, it makes the test more convoluted and slightly harder to debug, as you cannot directly see all the values being fed in (though there's always the option of generating test cases as either code or data, too). If you're doing some semi-complicated logic to generate your random test data, then there's also the chance that this code has a bug in it. Bugs in test code can be a pain, especially if developers immediate assume the bug is the production code.

Secondly, it is often impossible to be specific about the expected answer. If you know the answer based on the input, then there's a decent chance you're just aping the logic under test (think about it -- if the input is random, how do you know the expected output?) As a result, you may have to trade very specific asserts (the value should be x) for more general sanity-check asserts (the value should be between y and z).

Thirdly, unless there's a wide range of inputs and outputs, you can often cover the same range using well chosen values in a standard unit tests with less complexity. E.g. pick the numbers -max, (-max + 1), -2, -1, 0, 1, 2, max-1, max. (or whatever is interesting for the algorithm).

Upsides

When done well with the correct target, these tests can provide a very valuable complementary testing pass. I've seen quite a few bits of code that, when hammered by randomly generated test inputs, buckled due to unforeseen edge cases. I sometimes add an extra integration testing pass that generates a shedload of test cases.

Additional tricks

If one of your random tests fails, isolate the 'interesting' value and promote it into a standalone unit test to ensure that you can fix the bug and it will never regress prior to checkin.

Answer 2

As others have suggested, it makes your test unreliable because you don't know what's going on inside of it. That means it might work for some cases, and not for others.

If you already have an idea of the range of values that you want to test, then you should either (1) create a different test for each value in the range, or (2) loop over the set of values and make an assertion on each iteration. A quick, rather silly, example...

for($i = 0; $i < 10; $i++)
  $this->assertEquals($i + 1, Math::addOne($i));

You could do something similar with character encodings. For example, loop over the ASCII character set and test all of those crazy characters against one of your text manipulation functions.

Answer 3

It is much better to have unit tests that are 100% repeatable and include all the edge cases. For example, test zero, negatives, positives, numbers too big, numbers too small, etc. If you want to include tests with random values in addition to all the edge cases and normal cases that would be fine. However, I'm not sure you would get much benefit out of the time spent. Having all the normal cases and edge cases should handle everything. The rest is "gravy".

Answer 4

Additional Downside that hasn't been mentioned as yet is that your tests may intermittently fail at random, especially when you randomly generate several test variables so they form a convoluted and sometimes untractable dependencies. See example here.

Debugging these is a right pain in the backside and sometimes is (next to) impossible.

Also, it's often hard to tell what your test actually tests (and if it tests anything at all).

Historically in my company we use random tests at multiple levels (Unit, Integration, SingleService Tests), and that seemed like a great idea initially - it saves you code, space and time allowing to test multiple scenarios in one test.

But increasingly that gets to be a sticky point in our development, when our (even historic and reliable in the past) test start failing at random - and fixing these is way labour-intensive.

Answer 5

They are random.

(Your test might randomly work, even if your code is broken.)

Answer 6

Also, you won't be able to reproduce tests many times over. A unit test should run exactly the same with given parameters.