How should one use Disruptor (Disruptor Pattern) to build real-world message systems?

Question

As the RingBuffer up-front allocates objects of a given type, how can you use a single ring buffer to process messages of various different types?

You can't create new object instances to insert into the ringBuffer and that would defeat the purpose of up-front allocation.

So you could have 3 messages in an async messaging pattern:

1. NewOrderRequest
2. NewOrderCreated
3. NewOrderRejected

So my question is how are you meant to use the Disruptor pattern for real-world messageing systems?

Thanks

Links: http://code.google.com/p/disruptor-net/wiki/CodeExamples

http://code.google.com/p/disruptor-net

http://code.google.com/p/disruptor

Answer 1

One approach (our most common pattern) is to store the message in its marshalled form, i.e. as a byte array. For incoming requests e.g. Fix messages, binary message, are quickly pulled of the network and placed in the ring buffer. The unmarshalling and dispatch of different types of messages are handled by EventProcessors (Consumers) on that ring buffer. For outbound requests, the message is serialised into the preallocated byte array that forms the entry in the ring buffer.

If you are using some fixed size byte array as the preallocated entry, some additional logic is required to handle overflow for larger messages. I.e. pick a reasonable default size and if it is exceeded allocate a temporary array that is bigger. Then discard it when the entry is reused or consumed (depending on your use case) reverting back to the original preallocated byte array.

If you have different consumers for different message types you could quickly identify if your consumer is interested in the specific message either by knowing an offset into the byte array that carries the type information or by passing a discriminator value through on the entry.

Also there is no rule against creating object instances and passing references (we do this in a couple of places too). You do lose the benefits of object preallocation, however one of the design goals of the disruptor was to allow the user the choice of the most appropriate form of storage.

Answer 2

There is a library called Javolution (http://javolution.org/) that let's you defined objects as structs with fixed-length fields like string[40] etc. that rely on byte-buffers internally instead of variable size objects... that allows the token ring to be initialized with fixed size objects and thus (hopefully) contiguous blocks of memory that allow the cache to work more efficiently.

We are using that for passing events / messages and use standard strings etc. for our business-logic.

Answer 3

Back to object pools.

The following is an hypothesis.

If you will have 3 types of messages (A,B,C), you can make 3 arrays of those pre-allocated. That will create 3 memory zones A, B, C.

It's not like there is only one cache line, there are many and they don't have to be contiguous. Some cache lines will refer to something in zone A, other B and other C.

So the ring buffer entry can have 1 reference to a common ancestor or interface of A & B & C.

The problem is to select the instance in the pools; the simplest is to have the same array length as the ring buffer length. This implies a lot of wasted pooled objects since only one of the 3 is ever used at any entry, ex: ring buffer entry 1234 might be using message B[1234] but A[1234] and C[1234] are not used and unusable by anyone.

You could also make a super-entry with all 3 A+B+C instance inlined and indicate the type with some byte or enum. Just as wasteful on memory size, but looks a bit worse because of the fatness of the entry. For example a reader only working on C messages will have less cache locality.

I hope I'm not too wrong with this hypothesis.