What are the downsides using random values in Unit Testing?

Question

I\'m talking about a large scales system, with many servers and non deterministic input in high capacity. When i say non deterministic i\'m talking about messages that are s

Answer 1

The results aren't repeatable, and depending on your tests, you may not know the specific conditions which caused the code to fail (thus making it tough to debug).

Answer 2

I believe that generating random input values can be a reliable testing technique when used together with equivalence partitioning. This means that, if you partition your input space and then randomly pick values from an equivalence class, then you are fine: same coverage (any of them, including statement, branch, all-uses etc). This under the assumption that your equivalence partitioning procedure is sound. Also, I would recommend boundary value analysis to be paired with equivalence partitioning and randomly generated inputs.

Finally, I would also recommend considering the TYPE of defects you want to detect: some testing techniques address specific types of defects, which might be hardly (and just by chance) detected by other techniques. An example: deadlock conditions.

In conclusion, I believe that generating random values is not a bad practice, in particular in some systems (e.g. web applications), but it only addresses a subset of existing defects (like any other technique) and one should be aware of that, so to complement his/her quality assurance process with the adequate set of activities.

Answer 3

You need to remember which random numbers you generated during verification.

Example.

Username= "username".rand();
Save_in_DB("user",Username); // To save it in DB
Verify_if_Saved("user",Username); 

Answer 4

Upsides: they reveal when your other tests have not covered all the invariants. Whether you want your CI server to run nondeterministic tests is another issue. Given how incredibly useful I've found https://www.artima.com/shop/scalacheck, I don't intend to do without it from now on. Let's say you're implementing a pattern-matching algorithm. Do you really know all the different corner cases? I don't. Randomized inputs may flush them out.

Answer 5

Randomizing unit tests is using a screwdriver to hammer in a nail. The problem is not that screwdrivers are bad; the problem is you're using the wrong tool for the job. The point of unit tests is to provide immediate feedback when you break something, so you can fix it right there.

Let's say you commit a change, which we'll call BadChange. BadChange introduces a bug, which your random tests will sometimes catch and sometimes not. This time, the tests don't catch it. BadChange is given the all-clear and goes into the code base.

Later, someone commits another change, GoodChange. GoodChange is one hundred percent fine. But this time, your random tests catch the bug introduced by BadChange. Now GoodChange is flagged as a problem, and the developer who wrote it will be going in circles trying to figure out why this innocuous change is causing issues.

Randomized testing is useful to constantly probe the whole application for issues, not to validate individual changes. It should live in a separate suite, and runs should not be linked to code changes; even if no one has made a change, the possibility remains that the random tests will stumble across some exotic bug that previous runs missed.