How should I unit test threaded code?

Question

I have thus far avoided the nightmare that is testing multi-threaded code since it just seems like too much of a minefield. I\'d like to ask how people have gone about test

Answer 1

Assuming under "multi-threaded" code was meant something that is

• stateful and mutable
• AND accessed/modified by multiple threads concurrently

In other words we are talking about testing custom stateful thread-safe class/method/unit - which should be a very rare beast nowadays.

Because this beast is rare, first of all we need to make sure that there are all valid excuses to write it.

Step 1. Consider modifying state in same synchronization context.

Today it is easy to write compose-able concurrent and asynchronous code where IO or other slow operations offloaded to background but shared state is updated and queried in one synchronization context. e.g. async/await tasks and Rx in .NET etc. - they are all testable by design, "real" Tasks and schedulers can be substituted to make testing deterministic (however this is out of scope of the question).

It may sound very constrained but this approach works surprisingly well. It is possible to write whole apps in this style without need to make any state thread-safe (I do).

Step 2. If manipulating of shared state on single synchronization context is absolutely not possible.

Make sure the wheel is not being reinvented / there's definitely no standard alternative that can be adapted for the job. It should be likely that code is very cohesive and contained within one unit e.g. with a good chance it is a special case of some standard thread-safe data structure like hash map or collection or whatever.

Note: if code is large / spans across multiple classes AND needs multi-thread state manipulation then there's a very high chance that design is not good, reconsider Step 1

Step 3. If this step is reached then we need to test our own custom stateful thread-safe class/method/unit.

I'll be dead honest : I never had to write proper tests for such code. Most of the time I get away at Step 1, sometimes at Step 2. Last time I had to write custom thread-safe code was so many years ago that it was before I adopted unit testing / probably I wouldn't have to write it with the current knowledge anyway.

If I really had to test such code (finally, actual answer) then I would try couple of things below

1. Non-deterministic stress testing. e.g. run 100 threads simultaneously and check that end result is consistent. This is more typical for higher level / integration testing of multiple users scenarios but also can be used at the unit level.

2. Expose some test 'hooks' where test can inject some code to help make deterministic scenarios where one thread must perform operation before the other. As ugly as it is, I can't think of anything better.

3. Delay-driven testing to make threads run and perform operations in particular order. Strictly speaking such tests are non-deterministic too (there's a chance of system freeze / stop-the-world GC collection which can distort otherwise orchestrated delays), also it is ugly but allows to avoid hooks.

Answer 2

I also had serious problems testing multi- threaded code. Then I found a really cool solution in "xUnit Test Patterns" by Gerard Meszaros. The pattern he describes is called Humble object.

Basically it describes how you can extract the logic into a separate, easy-to-test component that is decoupled from its environment. After you tested this logic, you can test the complicated behaviour (multi- threading, asynchronous execution, etc...)

Answer 3

I spent most of last week at a university library studying debugging of concurrent code. The central problem is concurrent code is non-deterministic. Typically, academic debugging has fallen into one of three camps here:

1. Event-trace/replay. This requires an event monitor and then reviewing the events that were sent. In a UT framework, this would involve manually sending the events as part of a test, and then doing post-mortem reviews.
2. Scriptable. This is where you interact with the running code with a set of triggers. "On x > foo, baz()". This could be interpreted into a UT framework where you have a run-time system triggering a given test on a certain condition.
3. Interactive. This obviously won't work in an automatic testing situation. ;)

Now, as above commentators have noticed, you can design your concurrent system into a more deterministic state. However, if you don't do that properly, you're just back to designing a sequential system again.

My suggestion would be to focus on having a very strict design protocol about what gets threaded and what doesn't get threaded. If you constrain your interface so that there is minimal dependancies between elements, it is much easier.

Good luck, and keep working on the problem.

Answer 4

(if possible) don't use threads, use actors / active objects. Easy to test.

Answer 5

I like to write two or more test methods to execute on parallel threads, and each of them make calls into the object under test. I've been using Sleep() calls to coordinate the order of the calls from the different threads, but that's not really reliable. It's also a lot slower because you have to sleep long enough that the timing usually works.

I found the Multithreaded TC Java library from the same group that wrote FindBugs. It lets you specify the order of events without using Sleep(), and it's reliable. I haven't tried it yet.

The biggest limitation to this approach is that it only lets you test the scenarios you suspect will cause trouble. As others have said, you really need to isolate your multithreaded code into a small number of simple classes to have any hope of thoroughly testing them.

Once you've carefully tested the scenarios you expect to cause trouble, an unscientific test that throws a bunch of simultaneous requests at the class for a while is a good way to look for unexpected trouble.

Update: I've played a bit with the Multithreaded TC Java library, and it works well. I've also ported some of its features to a .NET version I call TickingTest.

Answer 6

I handle unit tests of threaded components the same way I handle any unit test, that is, with inversion of control and isolation frameworks. I develop in the .Net-arena and, out of the box, the threading (among other things) is very hard (I'd say nearly impossible) to fully isolate.

Therefore, I've written wrappers that looks something like this (simplified):

public interface IThread
{
    void Start();
    ...
}

public class ThreadWrapper : IThread
{
    private readonly Thread _thread;
     
    public ThreadWrapper(ThreadStart threadStart)
    {
        _thread = new Thread(threadStart);
    }

    public Start()
    {
        _thread.Start();
    }
}
    
public interface IThreadingManager
{
    IThread CreateThread(ThreadStart threadStart);
}

public class ThreadingManager : IThreadingManager
{
    public IThread CreateThread(ThreadStart threadStart)
    {
         return new ThreadWrapper(threadStart)
    }
}

From there, I can easily inject the IThreadingManager into my components and use my isolation framework of choice to make the thread behave as I expect during the test.

That has so far worked great for me, and I use the same approach for the thread pool, things in System.Environment, Sleep etc. etc.