How can I read messages from a queue in parallel?

Question

Situation

We have one message queue. We would like to process messages in parallel and limit the number of simultaneously processed messages.

Our trial code

Answer 1

You should look at using Microsoft's Reactive Framework for this.

You code could look like this:

var query =
    from command1 in FromQueue(queue)
    from text in Observable.Start(() =>
    {
        Thread.Sleep(1000);
        return "id: " + command1.id + ", name: " + command1.name;
    })
    select text;

var subscription =
    query
        .Subscribe(text => Console.WriteLine(text));

This does all of the processing in parallel, and ensures that the processing is properly distributed across all cores. When one value ends another starts.

To cancel the subscription just call subscription.Dispose().

The code for FromQueue is:

static IObservable FromQueue(string serverQueue)
{
    return Observable.Create(observer =>
    {
        var responseQueue = Environment.MachineName + "\\Private$\\" + Guid.NewGuid().ToString();
        var queue = MessageQueue.Create(responseQueue);

        var frm = new System.Messaging.BinaryMessageFormatter();
        var srv = new MessageQueue(serverQueue);
        srv.Formatter = frm;
        queue.Formatter = frm;

        srv.Send("S " + responseQueue);

        var loop = NewThreadScheduler.Default.ScheduleLongRunning(cancel =>
        {
            while (!cancel.IsDisposed)
            {
                var msg = queue.Receive();
                observer.OnNext((T)msg.Body);
            }
        });

        return new CompositeDisposable(
            loop,
            Disposable.Create(() =>
            {
                srv.Send("D " + responseQueue);
                MessageQueue.Delete(responseQueue);
            })
        );
    });
}

Just NuGet "Rx-Main" to get the bits.

---

In order to limit the concurrency you can do this:

int maxConcurrent = 2;
var query =
    FromQueue(queue)
        .Select(command1 => Observable.Start(() =>
        {
            Thread.Sleep(1000);
            return "id: " + command1.id + ", name: " + command1.name;
        }))
        .Merge(maxConcurrent);