The concurrent implementation of a publisher/subscriber pattern

问题

I want to implement a variety of of a publisher/subscriber pattern using Java and currently running out of ideas.

There is 1 publisher and N subscribers, the publisher publish objects then each subscriber need to process the each of the objects once and only once in the correct order. The publisher and each subscriber run in their own thread.

In my original implementation, each subscriber has its own blocking queue, and the publisher put objects into each of the subscriber's queue. This works ok but the publisher will be blocked if any of the subscriber's queue is full. This leads to degration of performance as each subscriber takes different time in processing the object.

Then in another implementation, the publisher hold the object in its own queue. Along with the object, an AtomicInteger counter is associated with it with the number of subscribers out there. Each subscriber then peek the queue and decrease the counter, and remove it from the queue when the counter reach zero.

In this way the publisher is free from blocking but now the subscriber will need to wait for each other to process the object, removing the object from the queue, before the next object can be peeked.

Is there any better way to do this? I assume this should be a quite common pattern.

回答1:

Your "many queues" implementation is the way to go. I don't think you need to necessarily be concerned with one full queue blocking the producer, because the overall time to completion won't be affected. Let's say you have three consumers, two consume at a rate of 1 per second and the third consumes at a rate of 1 per five seconds, meanwhile the producer produces at a rate of 1 per two seconds. Eventually the third queue will fill up, and so the producer will block on it and will also stop putting items in the first and second queues. There are ways around this, but they're not going to change the fact that the third consumer will always be the bottleneck. If you're producing/consuming 100 items, then this will take at least 500 seconds because of the third consumer (5 seconds times 100 items), and this will be the case even if the first and second consumers finish after 200 seconds (because you've done something clever to allow the producer to continue to fill their queues even after the third queue is full) or if they finish after 500 seconds (because the producer blocked on the third queue).

回答2:

Definately

each subscriber has its own blocking queue, and the publisher put objects into each of the subscriber's queue.`

this is the way to go. you can use threaded approach to put it in queue... so if one queue is full publisher will not wait..

for example.

s1 s2 s3 are subscribers and addToQueue is method in each subscriber which adds to corrosponding queue. The addQueue Method is which waits till queue is non empty .. so call to addQueue will be a blocking call ideally synchronised code...

Then in publisher you can do something similar to below code

NOTE: code might not be in working condition as it is.. but should give you idea.

List<subscriber> slist;// Assume its initialised
public void publish(final String message){

    for (final subscriber s: slist){


          Thread t=new Thread(new Runnable(){
             public void run(){
                s.addToQueue(message);
             }
           });

      t.start();
    }

}

回答3:

There is 1 publisher and N subscribers, the publisher publish objects then each subscriber need to process the each of the objects once and only once in the correct order. The publisher and each subscriber run in their own thread.

I would change this architecture. I initially considered the queue per subscriber but I don't like that mechanism. For example, if the first subscriber takes longer to run, all of the jobs will end up in that queue and you will only be doing 1 thread of work.

Since you have to run the subscribers in order, I'd have a pool of threads which run each message through all of the subscribers. The calls to the subscribers will need to be reentrant which may not be possible.

So you would have a pool of 10 threads (let's say) and each one dequeues from the publisher's queue, and does something like the following:

public void run() {
    while (!shutdown && !Thread.currentThread().isInterrupted()) {
        Article article = publisherQueue.take();
        for (Subscriber subscriber : subscriberList) {
           subscriber.process(article);
        }
    }
}

来源：https://stackoverflow.com/questions/16861729/the-concurrent-implementation-of-a-publisher-subscriber-pattern