Unit test for thread safe-ness?

Question

I\'ve written a class and many unit test, but I did not make it thread safe. Now, I want to make the class thread safe, but to prove it and use TDD, I want to write some fa

Answer 1

Though it's not as elegant as using a tool like Racer or Chess, I have used this sort of thing for testing for thread safety:

// from linqpad

void Main()
{
    var duration = TimeSpan.FromSeconds(5);
    var td = new ThreadDangerous(); 

    // no problems using single thread (run this for as long as you want)
    foreach (var x in Until(duration))
        td.DoSomething();

    // thread dangerous - it won't take long at all for this to blow up
    try
    {           
        Parallel.ForEach(WhileTrue(), x => 
            td.DoSomething());

        throw new Exception("A ThreadDangerException should have been thrown");
    }
    catch(AggregateException aex)
    {
        // make sure that the exception thrown was related
        // to thread danger
        foreach (var ex in aex.Flatten().InnerExceptions)
        {
            if (!(ex is ThreadDangerException))
                throw;
        }
    }

    // no problems using multiple threads (run this for as long as you want)
    var ts = new ThreadSafe();
    Parallel.ForEach(Until(duration), x => 
        ts.DoSomething());      

}

class ThreadDangerous
{
    private Guid test;
    private readonly Guid ctrl;

    public void DoSomething()
    {           
        test = Guid.NewGuid();
        test = ctrl;        

        if (test != ctrl)
            throw new ThreadDangerException();
    }
}

class ThreadSafe
{
    private Guid test;
    private readonly Guid ctrl;
    private readonly object _lock = new Object();

    public void DoSomething()
    {   
        lock(_lock)
        {
            test = Guid.NewGuid();
            test = ctrl;        

            if (test != ctrl)
                throw new ThreadDangerException();
        }
    }
}

class ThreadDangerException : Exception 
{
    public ThreadDangerException() : base("Not thread safe") { }
}

IEnumerable<ulong> Until(TimeSpan duration)
{
    var until = DateTime.Now.Add(duration);
    ulong i = 0;
    while (DateTime.Now < until)
    {
        yield return i++;
    }
}

IEnumerable<ulong> WhileTrue()
{
    ulong i = 0;
    while (true)
    {
        yield return i++;
    }
}

The theory is that if you can cause a thread dangerous condition consistently to occur in a very short amount of time, you should be able to bring about thread safe conditions and verify them by waiting a relatively large amount of time without observing state corruption.

I do admit that this may be a primitive way of going about it and may not help in complex scenarios.

Answer 2

you'll have to construct a test case for each concurrency scenario of concern; this may require replacing efficient operations with slower equivalents (or mocks) and running multiple tests in loops, to increase the chance of contentions

without specific test cases, it is difficult to propose specific tests

some potentially useful reference material:

• Oren Ellenbogen's Blog
• Vertical Slice Blog
• possible duplicate SO question

Answer 3

Note that Dror's answer does not explicitly say this, but at least Chess (and probably Racer) work by running a set of threads through all their possible interleavings to get repreoducible errors. They do not just run the threads for a while hoping that if there is an error it will happen by coincidence.

Chess for example will run through all interleavings and then give you a tag string that represents the interleaving that a deadlock was found on so that you can attribute your tests with the specific interleavings that are interesting from a deadlocking perspective.

I do not know the exact internal workings of this tool, and how it maps these tag strings back to code that you may be changing to fix a deadlock, but there you have it... I am actually really looking forward to this tool (and Pex) becoming part of the VS IDE.

Answer 4

Here's my approach. This test is not concerned with deadlocks, it's concerned with consistency. I'm testing a method with a synchronized block, with code that looks something like this:

synchronized(this) {
  int size = myList.size();
  // do something that needs "size" to be correct,
  // but which will change the size at the end.
  ...
}

It's tough to produce a scenario that will reliably produce a thread conflict, but here's what I did.

First, my unit test created 50 threads, launched them all at the same time, and had them all call my method. I use a CountDown Latch to start them all at the same time:

CountDownLatch latch = new CountDownLatch(1);
for (int i=0; i<50; ++i) {
  Runnable runner = new Runnable() {
    latch.await(); // actually, surround this with try/catch InterruptedException
    testMethod();
  }
  new Thread(runner, "Test Thread " +ii).start(); // I always name my threads.
}
// all threads are now waiting on the latch.
latch.countDown(); // release the latch
// all threads are now running the test method at the same time.

This may or may not produce a conflict. My testMethod() should be capable of throwing an exception if a conflict occurs. But we can't yet be sure that this will generate a conflict. So we don't know if the test is valid. So here's the trick: Comment out your synchronized keyword(s) and run the test. If this produces a conflict, the test will fail. If it fails without the synchronized keyword, your test is valid.

That's what I did, and my test didn't fail, so it was not (yet) a valid test. But I was able to reliably produce a failure by putting the code above inside a loop, and running it 100 times consecutively. So I call the method 5000 times. (Yes, this will produce a slow test. Don't worry about it. Your customers won't be bothered by this, so you shouldn't either.)

Once I put this code inside an outer loop, I was able to reliably see a failure on about the 20th iteration of the outer loop. Now I was confident the test was valid, and I restored the synchronized keywords to run the actual test. (It worked.)

You may discover that the test is valid on one machine and not on another. If the test is valid on one machine and your methods pass the test, then it's presumably thread-safe on all machines. But you should test for validity on the machine that runs your nightly unit tests.

Answer 5

I have seen people try to test this with standard unittests as you yourself propose. The tests are slow, and have so far failed to identify a single of the concurrency problems our company struggles with.

After many failures, and despite my love for unittests, I have come to accept that errors in concurrency is not one of unittests strengths. I usually encourage analysis and review in favour of unittests for classes where concurrency is a subject. With total overview of the system it is in many cases possible to prove/falsify claims of thread safety.

Anyhow I would love for someone to give me something that might point to the opposite, so I watch this question closely.

Answer 6

The problem is that most of the multi-threading issues, like race conditions, are non-deterministic by their nature. They can depend on hardware behavior which you can't possibly emulate or trigger.

That means, even if you make tests with multiple threads, they will not be consistently failing if you have a defect in your code.