How should I unit test threaded code?

Question

I have thus far avoided the nightmare that is testing multi-threaded code since it just seems like too much of a minefield. I\'d like to ask how people have gone about test

Answer 1

For J2E code, I've used SilkPerformer, LoadRunner and JMeter for concurrency testing of threads. They all do the same thing. Basically, they give you a relatively simple interface for administrating their version of the proxy server, required, in order to analyze the TCP/IP data stream, and simulate multiple users making simultaneous requests to your app server. The proxy server can give you the ability to do things like analyze the requests made, by presenting the whole page and URL sent to the server, as well as the response from the server, after processing the request.

You can find some bugs in insecure http mode, where you can at least analyze the form data that is being sent, and systematically alter that for each user. But the true tests are when you run in https (Secured Socket Layers). Then, you also have to contend with systematically altering the session and cookie data, which can be a little more convoluted.

The best bug I ever found, while testing concurrency, was when I discovered that the developer had relied upon Java garbage collection to close the connection request that was established at login, to the LDAP server, when logging in. This resulted in users being exposed to other users' sessions and very confusing results, when trying to analyze what happened when the server was brought to it's knees, barely able to complete one transaction, every few seconds.

In the end, you or someone will probably have to buckle down and analyze the code for blunders like the one I just mentioned. And an open discussion across departments, like the one that occurred, when we unfolded the problem described above, are most useful. But these tools are the best solution to testing multi-threaded code. JMeter is open source. SilkPerformer and LoadRunner are proprietary. If you really want to know whether your app is thread safe, that's how the big boys do it. I've done this for very large companies professionally, so I'm not guessing. I'm speaking from personal experience.

A word of caution: it does take some time to understand these tools. It will not be a matter of simply installing the software and firing up the GUI, unless you've already had some exposure to multi-threaded programming. I've tried to identify the 3 critical categories of areas to understand (forms, session and cookie data), with the hope that at least starting with understanding these topics will help you focus on quick results, as opposed to having to read through the entire documentation.

Answer 2

Pete Goodliffe has a series on the unit testing of threaded code.

It's hard. I take the easier way out and try to keep the threading code abstracted from the actual test. Pete does mention that the way I do it is wrong but I've either got the separation right or I've just been lucky.

Answer 3

I have had the unfortunate task of testing threaded code and they are definitely the hardest tests I have ever written.

When writing my tests, I used a combination of delegates and events. Basically it is all about using PropertyNotifyChanged events with a WaitCallback or some kind of ConditionalWaiter that polls.

I am not sure if this was the best approach, but it has worked out for me.

Answer 4

Awaitility can also be useful to help you write deterministic unit tests. It allows you to wait until some state somewhere in your system is updated. For example:

await().untilCall( to(myService).myMethod(), greaterThan(3) );

or

await().atMost(5,SECONDS).until(fieldIn(myObject).ofType(int.class), equalTo(1));

It also has Scala and Groovy support.

await until { something() > 4 } // Scala example

Answer 5

The following article suggests 2 solutions. Wrapping a semaphore (CountDownLatch) and adds functionality like externalize data from internal thread. Another way of achieving this purpose is to use Thread Pool (see Points of Interest).

Sprinkler - Advanced synchronization object

Answer 6

Concurrency is a complex interplay between the memory model, hardware, caches and our code. In the case of Java at least such tests have been partly addressed mainly by jcstress. The creators of that library are known to be authors of many JVM, GC and Java concurrency features.

But even this library needs good knowledge of the Java Memory Model specification so that we know exactly what we are testing. But I think the focus of this effort is mircobenchmarks. Not huge business applications.