Multiplex on queue.Queue?

Question

How can I go about \"selecting\" on multiple queue.Queue\'s simultaneously?

Golang has the desired feature with its channels:

select {
case i1 = <-c1:         


        

Answer 1

from queue import Queue

# these imports needed for example code
from threading import Thread
from time import sleep
from random import randint

class MultiQueue(Queue):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.queues = []

    def addQueue(self, queue):
        queue.put = self._put_notify(queue, queue.put)
        queue.put_nowait = self._put_notify(queue, queue.put_nowait)
        self.queues.append(queue)

    def _put_notify(self, queue, old_put):
        def wrapper(*args, **kwargs):
            result = old_put(*args, **kwargs)
            self.put(queue)
            return result
        return wrapper

if __name__ == '__main__':
    # an example of MultiQueue usage

    q1 = Queue()
    q1.name = 'q1'
    q2 = Queue()
    q2.name = 'q2'
    q3 = Queue()
    q3.name = 'q3'

    mq = MultiQueue()
    mq.addQueue(q1)
    mq.addQueue(q2)
    mq.addQueue(q3)

    queues = [q1, q2, q3]
    for i in range(9):
        def message(i=i):
            print("thread-%d starting..." % i)
            sleep(randint(1, 9))
            q = queues[i%3]
            q.put('thread-%d ending...' % i)
        Thread(target=message).start()

    print('awaiting results...')
    for _ in range(9):
        result = mq.get()
        print(result.name)
        print(result.get())

Rather than try to use the .get() method of several queues, the idea here is to have the queues notify the MultiQueue when they have data ready -- sort of a select in reverse. This is achieved by having MultiQueue wrap the various Queue's put() and put_nowait() methods so that when something is added to those queues, that queue is then put() into the the MultiQueue, and a corresponding MultiQueue.get() will retrieve the Queue that has data ready.

This MultiQueue is based on the FIFO Queue, but you could also use the LIFO or Priority queues as the base depending on your needs.