Unit testing a multithreaded application?

Question

Does anyone have any advice for a consistent way to unit test a multithreaded application? I have done one application where our mock \"worker threads\" had a thread.sleep

Answer 1

I've come across a research product, called Microsoft Chess. It's specifically designed for non-deterministic testing of multithreaded applications. Downside so far is that it is integrated into VS.

Answer 2

My advice would be not to rely on unit tests to detect concurrency issues for several reasons:

• Lack of reproducibility: the tests will fail only once in a while, and won't be really helpful to pinpoint the problems.
• Erratic failing build will annoy everybody in the team - because the last commit will always be wrongly suspected for being the cause of the failing build.
• Deadlocks when encountered are likely to freeze the build until the execution timeout is encountered which can significantly slow down the build.
• The build environment is likely to be a single CPU environment (think build being run in a VM) where concurrency issues may never happen - no matter how much sleeping time is set.
• It defeats somehow the idea of having simple, isolated units of validating code.

Answer 3

Like GUI Unit-testing, this is a bit of a Waterloo for Automated tests. True threads are by definition.. unpredictable.. they'll interfere in ways that can't be pre-determined. Hence writing true tests is difficult if not impossible.

However you have some company with this... I suggest searching through the archives of the testdrivendevelopment yahoo group. I remember some posts in the vicinity.. Here's one of the newer ones. (If someone would be kind enough to strafe and paraphrase.. that would be great. I'm too sleepy.. Need to LO SO)

Answer 4

It's important to test multi-threaded code on a multi-processor machine. A dual-core machine may not be sufficient. I've seen deadlocks occur on a 4 processor machine that did not occur on a dual-core single processor. Then you need to create a stress test based on a client program that spawns many threads and makes multiple requests against the target application. It helps if the client machine is multi-processor as well so there is more load on the target application.

Answer 5

If you have to test that a background thread does something, a simple technique I find handy is to to have a WaitUntilTrue method, which looks something like this:

bool WaitUntilTrue(Func<bool> func,
              int timeoutInMillis,
              int timeBetweenChecksMillis)
{
    Stopwatch stopwatch = Stopwatch.StartNew();

    while(stopwatch.ElapsedMilliseconds < timeoutInMillis)
    {
        if (func())
            return true;
        Thread.Sleep(timeBetweenChecksMillis);
    }   
    return false;
}

Used like this:

volatile bool backgroundThreadHasFinished = false;
//run your multithreaded test and make sure the thread sets the above variable.

Assert.IsTrue(WaitUntilTrue(x => backgroundThreadHasFinished, 1000, 10));

This way you don't have to sleep your main testing thread for a long time to give the background thread time to finish. If the background doesn't finish in a reasonable amount of time, the test fails.

Answer 6

I don't think that unit tests are an effective way to find threading bugs, but they can be a good way to demonstrate a known threading bug, isolate it, and test your fix for it. I've also used them to test the basic features of some coordinating class in my application, like a blocking queue for example.

I ported the multithreadedTC library from Java to .NET and called it TickingTest. It lets you start up several threads from a unit test method and coordinate them. It doesn't have all the features of the original, but I've found it useful. The biggest thing it's missing is the ability to monitor threads that are started during the test.