How to make worker threads quit after work is finished in a multithreaded producer-consumer pattern?

Question

I am trying to implement a multithreaded producer-consumer pattern using Queue.Queue in Python 2.7. I am trying to figure out how to make the consumers, i.e. the worker threads,

Answer 1

In addition to @DanielSanchez excellent answer, I propose to actually rely on a similar mechanism as a Java CountDownLatch.

The gist being,

• you create a latch that will open only after a certain counter went down,

• when the latch is opened, the thread(s) waiting on it will be allowed to proceed with their execution.

• I made an overly simple example, check here for a class like example of such a latch:

  import threading
  import Queue
  import time
  
  WORKER_COUNT = 3
  latch = threading.Condition()
  count = 3
  
  def wait():
      latch.acquire()
      while count > 0:
          latch.wait()
      latch.release()
  
  def count_down():
      global count
      latch.acquire()
      count -= 1
      if count <= 0:
          latch.notify_all()
      latch.release()
  
  def worker(n, q):
      # n - Worker ID
      # q - Queue from which to receive data
      while True:
          data = q.get()
          print 'worker', n, 'got', data
          time.sleep(1)  # Simulate noticeable data processing time
          q.task_done()
          if data == -1: # -1 is used to indicate that the worker should stop
              # Requeue the exit indicator.
              q.put(-1)
              # Commit suicide.
              count_down()
              print 'worker', n, 'is exiting'
              break
  
  # master() sends data to worker() via q.  
  
  def master():
      q = Queue.Queue()
  
      # Create 3 workers.
      for i in range(WORKER_COUNT):
          t = threading.Thread(target=worker, args=(i, q))
          t.start()
  
      # Send 10 items to work on.
      for i in range(10):
          q.put(i)
          time.sleep(0.5)
  
      # Send an exit indicator for all threads to consume.
      q.put(-1)
      wait()
      print 'done'
  
  master()