Processing a queue of items asynchronously in C#

Question

I am trying to create a system that processes a queue of work. The system has the following specifications:

1. The system has two components, a work assigner and a wo

Answer 1

You are on the right path. You could use MSMQ and a multi threaded C# service. I got my start writing C# multi threaded services with this article. The article is getting some age on it but the principles just haven't changed so it's relevant. It's easy to understand and better yet it explores both approaches you've suggest. Feel free to email me if you need additional assistance.

Answer 2

I don't know how long your tasks will be running, but it seems that the best thing to do would be to use ThreadPool. Furthermore, I would use, and actually have used, only one central queue - that alone will remove some complexity. I have one Thread that handles the queue and does an action on the item in your case it would be to queue a task.

As for making the queue threadsafe, there is a ConcurrentQueue in System.Collections.Concurrent for that very purpose (msdn, benchmark vs locking queue).

Now, throw in a BlockingCollection (msdn) and you have all you need.

        BlockingCollection<Packet> sendQueue = new BlockingCollection<Packet>(new ConcurrentQueue<Packet>());
        while (true)
        {
            var packet = sendQueue.Take(); //this blocks if there are no items in the queue.
            ThreadPool.QueueUserWorkItem(state =>
            {
               var data = (Packet)state;
               //do whatever you have to do
            }, packet );
        }

and somewhere there is something that sendQueue.Add(packet);

To sum up,

1. One queue for all "workers"
2. One thread that dequeues from the queue and passes it to the ThreadPool.

I think that's it.

ps: if you have to control the amount of threads, use "Smart Thread Pool" as suggested by josh3736

Answer 3

Use a thread pool. Here's one that handles the queueing of work items and dispatching them across the thread pool.