Sharing a Queue instance between different modules

问题

I am new to Python and I would like to create what is a 'global static variable', my thread-safe and process-safe queue, between threads/processes created in different modules. I read from the doc that the concept of a global variable is created using a third module, that I will call as cfg, which defines and initializes my global queue. I have a problem sharing the instance of this objects between my modules because I tried to print the repr() function over the shared queue imported from the cfg module, inside my other modules that import it, and it shows that they are different instances. It seems that every time I try to import a module a new instance is created and passed to the module who imports it.

Main.py:

import GatewayManager
if __name__ == '__main__':
    GatewayManager.initialize()
    doSomething()

GatewayManager.py:

import multiprocessing
import queue
import threading

def initialize():
    # Multiprocessing or Threading
    global isMonoCPU
    isMonoCPU = multiprocessing.cpu_count() == 1

    global sharedQueue
    sharedQueue = multiprocessing.Queue() if not isMonoCPU else queue.Queue()
    print("gateway: ", sharedQueue.__repr__())

otherModules.py:

import GatewayManager

# Some module write on the queue
GatewayManager.sharedQueue.put(variable)

# Some read from the queue
GatewayManager.sharedQueue.get()
print("driver: ", GatewayManager.sharedQueue.__repr__())

回答1:

Here:

# GatewayManager.py:

...

def initialize():
    global sharedQueue
    # ...
    sharedQueue = multiprocessing.Queue()
    # ...

Your GatewayManager module don't have a sharedQueue attribute until it's initialize() function is called., so if any other module tries to use GatewayManager.sharedQueue before GatewayManager.initialize() has been called then of course you'll get this error. And since GatewayManager.initialize() blindly rebinds sharedQueue on each call, if you call it again from another module then you lose the already created queue and gets a new one.

What you want is to make sure your sharedqueue is created once and only once, and that it will be created whatever happens. The solution (well, one solution at least - but it's a known working solution) here is to proxy all GatewayManager.sharedQueue.whatever access thru functions that will take care of initializing the queue if and when needed.

# gateway_manager.py

class _QueueProxy(object):
    def __init__(self):
        self._queueimp = None

    @property
    def _queue(self):
        if self._queueimp is None:
            isMonoCPU = multiprocessing.cpu_count() == 1
            self._queueimp = queue.Queue() if isMonoCPU else multiprocessing.Queue() 

        return self._queueimp

    def get(self, *args, **kw):
        return self._queue.get(*args, **kw)

    def put(self, *args, **kw):
        return self._queue.put(*args, **kw)

   # etc... only expose public methods and attributes of course


# and now our `shared_queue` instance    
shared_queue = _QueueProxy()

And now you can safely (well almost - the queue creation is not atomic so you may have race conditions) use gateway_manager.shared_queue from any module without having to care about initialization.

Of course if you have two distinct processes (I'm not talking about multiprocessing.Process here) you will still have two distinct queues but I assume you already understood this (and if not please read Jean-Paul's answer).

回答2:

A multiprocessing.Queue is shared between the process that creates it (let's call it "Parent") and processes created by Parent (let's call them "Children").

Here is an example of some processes that do not have this relationship:

$ python myprogram.py &
$ python myprogram.py &

The shell is the Parent of these two Children. The shell did not create the multiprocessing.Queue, though, so it will not be shared by the two children. Instead, they will each create their own. This may be shared with their children but not with each other.

You can easily observe this behavior:

$ cat queuedemo.py 
from time import sleep
from os import getpid
from sys import argv

from multiprocessing import Queue

q = Queue()

if argv[1:]:
    q.put(getpid())
    sleep(60)
else:
    print(getpid(), q.get())
exarkun@baryon:/tmp/queue$ python queuedemo.py foo & python queuedemo.py 
[1] 28249

The second process never manages to read anything from the queue. However, if you give the two process the Parent-Child relationship...

$ cat queuedemo.py 
from os import getpid

from multiprocessing import Queue
from multiprocessing.process import Process

q = Queue()
q.put(getpid())

def child():
    print(getpid(), q.get())

p = Process(target=child)
p.start()
p.join()
exarkun@baryon:/tmp/queue$ python queuedemo.py 
(28469, 28467)
exarkun@baryon:/tmp/queue$

Notice that the q.get() call succeeds and that the pid put into the queue is different from the pid of the process that gets it out.

Somewhat necessarily, this also extends to processes with Parent-Descendant and Sibling relationships.

So:

Globals are only shared within a single process
The multiprocessing module provides tools to share state between processes that are properly related to each other.

If you want to share state between processes without this relationship, there are a variety of other options - and the best one will depend a bit more on what kind of state you have to share and what your sharing patterns look like (neither of which you've included in your question).

来源：https://stackoverflow.com/questions/48705408/sharing-a-queue-instance-between-different-modules

标签

python

multithreading

global