Any satisfactory approaches to unit testing thread safety in Java?

Question

I am looking at improving a package that I believe not to be threadsafe when its input is shared between multiple worker threads. According to TDD principles, I should write so

Answer 1

There are two basic problems you have to solve. The first is this: how do you generate a thread clash? Second, how do you validate your test?

The first one is straightforward. Use a big hammer. Write some code that's capable of detecting the case where one thread steps on another, and run that code 50 times or so on 50 consecutive threads. You can do this with a countdown latch:

public void testForThreadClash() {
  final CountDownLatch latch = new CountDownLatch(1);
  for (int i=0; i<50; ++i) {
    Runnable runner = new Runnable() {
      public void run() {
        try {
          latch.await();
          testMethod();
        } catch (InterruptedException ie) { }
      }
    }
    new Thread(runner, "TestThread"+i).start();
  }
  // all threads are waiting on the latch.
  latch.countDown(); // release the latch
  // all threads are now running concurrently.
}

Your testMethod() has to generate a situation where, without the synchronized block, some thread will step on another thread's data. It should be able to detect this case and throw an exception. (The code above doesn't do this. The exception will most likely be generated on one of the test threads, and you need to put in a mechanism that will detect it on the main thread, but that's a separate issue. I left it out for simplicity, but you can do this with a second latch, which the test can prematurely clear on an exception.)

The second question is trickier, but there's a simple solution. The question is this: How do you know that your hammer will generate a clash? Of course, your code has synchronized blocks to prevent a clash, so you can't answer the question. But you can. Just remove the synchronized keywords. Since it's the synchronized keyword that makes your class threadsafe, you can just remove them and re-run the test. If the test is valid, it will now fail.

When I first wrote the code above, it turned out to be invalid. I never saw a clash. So it's not (yet) a valid test. But now we know how to get a clear answer to the second question. We're getting the wrong answer, but we can now tinker with the test to generate the failure we're looking for. Here's what I did: I just ran the test 100 times in a row.

for (int j=0; j<100; ++j) {
  testForThreadClash();
}

Now my test failed reliably on about the 20th iteration or so. This confirms my test is valid. I can now restore the synchronized keyword and rerun the test, confident that it will tell me if my class is threadsafe.

Answer 2

Forget getting good results by testing for concurrency problems. Attempt to reduce synchronization and make the problem smaller. Then use as high as possible library support to do synchronization. And only if you really have then try to handle the concurrency yourself. When you know each worker does its work correct and all your thinking tell you that you have the concurrency problem licked then generate some interesting load. Unittest frameworks and their extensions can do that job, but know that you are not testing any units anymore. (Remember, you already had that part covered)

If your concurrency model gets complicated check out tools suited for that like SPIN.

Answer 3

Java Concurrency in Practice has some great information about how to write tests for concurrency issues. However they are not true unit tests. It is nearly impossible to write a true unit test for a concurrency issue.

Basically it boils down to this. Create a bunch of test threads and start them. Each thread should

• wait for a count down latch
• repeatedly call some method that modifies the mutable state in question
• count down on a second latch and exit

The junit thread creates all the threads and starts them, then counts down on the first latch once to let them all go, then waits for the second latch, then makes some assertions about the mutable state.

Even more so than other types of bugs, it's easier to write a failing unit test for a concurrency bug after you have found the bug.

Answer 4

In some cases, I've found I can force a particular problematic interleaving of calls to a potentially un-thread-safe class by using multiple threads that synchronize with each other, perhaps with the use of CountDownLatch or some such concurrency mechanism. Sometimes this just doesn't work however, for instance if you're trying to test what happens if two threads are in the same method at the same time.

Here's an interesting article (don't know much about the tool, though) : http://today.java.net/pub/a/today/2003/08/06/multithreadedTests.html

Answer 5

There is nothing wrong with unit testing multi-threaded code if threading is the point of the code you're testing (think concurrent data structures). The general approach is to block the main test thread, perform assertions in separate threads, capturing and re-throwing any failed assertions in the main thread, else unblock the main test thread so the test can complete. It's pretty straightforward to do with ConcurrentUnit:

@Test
public void shouldDeliverMessage() throws Throwable {
  final Waiter waiter = new Waiter();

  messageBus.registerHandler(message -> {
    // Called on separate thread
    waiter.assertEquals(message, "foo");

    // Unblocks the waiter.await call
    waiter.resume();
  };

  messageBus.send("foo");

  // Wait for resume() to be called
  waiter.await(1000);
}

The key here is that any failed assertions in any thread will be re-thrown in the main thread by the waiter, allowing the test to pass or fail as it should.