问题
Imagine the following code:
import threading, zmq, time
context = zmq.Context()
receivers = []
poller = zmq.Poller()
def thread_fn(number: int):
sender = context.socket(zmq.PUSH)
sender.connect("tcp://localhost:%d" % (6666 + number))
for i in range(10):
sender.send_string("message from thread %d" % number)
for i in range(3):
new_receiver = context.socket(zmq.PULL)
new_receiver.bind("tcp://*:%d" % (6666 + i))
poller.register(new_receiver, zmq.POLLIN)
receivers.append(new_receiver)
threading.Thread(target=lambda: thread_fn(i), daemon=True).start()
while True:
try:
socks = dict(poller.poll())
except KeyboardInterrupt:
break
for i in range(3):
if receivers[i] in socks:
print("%d: process message %s" % (i, receivers[i].recv_string()))
time.sleep(0.2) # 'process' the data
The threads send some messages without interruption which arrive in some random order at the corresponding PULL-sockets where they get 'processed'.
Note: usually you would connect to one PULL-socket but this example intends to provide more than one receiving socket.
Output is:
0: process message message from thread 0
1: process message message from thread 1
0: process message message from thread 0
1: process message message from thread 1
2: process message message from thread 2
0: process message message from thread 0
1: process message message from thread 1
2: process message message from thread 2
....
Now I want to read from all sockets like in the example but I'd like to prioritise one socket.
I.e.: I want the output to be:
0: process message message from thread 0 <-- socket 0 processed first
0: process message message from thread 0
0: process message message from thread 0
0: process message message from thread 0
0: process message message from thread 0
1: process message message from thread 1
1: process message message from thread 1
2: process message message from thread 2
1: process message message from thread 1
2: process message message from thread 2
....
Of course I can just poll the sockets separately with timeout=0 but I want to be sure ZeroMQ doesn't do this for me already.
So the questions are:Q1:
Is there another way ( except the built-in .poll( timeout ) )
to make sure I've read messages from one socket first
before waiting for messages on the other sockets?
Q2:
Is there a known best practice to do it manually?
回答1:
Welcome to the Wild Worlds of (managed)-chaos,
A1:YesTL;DRcheckzmq.select()in recent API / python wrapperA2:YesTL;DRa must do part of the rigorous dependable system design
For the sake of Q2, the system design ought remain flexible, not only to handle the said prioritisation segmentation, but also to provide serious means for a robust handling of remote failures to comply with ( just optimistically ) expected modus operandi.
What does that mean?
If the postulated behaviour were implemented with some trivial and naive serial alignment of easy to implement principal syntax-constructs' sections alike this idea:
# --------------------------------------------------------
# FIRST scan all HI-PRIO socket(s) for incoming messages:
while true:
# process 'em first, based on a ZeroMQ-socket's behaviour-fixed ordering
...
break
# --------------------------------------------------------
# NEXT scan all LO-PRIO socket(s) for incoming messages:
while true:
# process 'em, again, based on a ZeroMQ-socket's behaviour-fixed ordering
...
break
any benefits your system architecture strives to create are lost in the very moment you forget to have a robust Plan B - how to handle blocking states, lost messages, dead counterparty FSA-process, DoS-attack, just a faulty remote-NIC that suddenly sprays your inbound interface with spurious and massive flow of bytes, all nightmares that may and do appear out of your controls.
This means, carefully plan for how to survive a case, the first group of sockets started to "feed" your receiver with so many messages ( processing tasks ), that you "can" never exit from the HI-PRIO section.
If still not getting the point, let me remind the great system design, introduced for this very purpose into Apollo Guidance Computer (AGC) software by an MIT team headed by Ms. Margaret HAMILTON, which did survive such "infinite-attack" of events, that were not anticipated by engineering guys, but which did happen during real-life, the worse, during the landing of the Eagle ( the Lunar Module ) on the Moon - the second most critical phase of the whole journey "there and back".
It is not any overhype to state that the smart design from Ms. Hamilton's team did save both the very moment and the whole glory of the U.S. most prestigious Apollo Programme.
Every responsible design has to be planned so as to survive this.
The current ZeroMQ wrapper for python provides for such purpose a tool, a Poller() class, that -- once due care is taken -- may save both your design targets and provide a space for adding a reliability-motivated functions, incl. a fallback escape strategies to be taken on colliding priorities/resources situations.
# ------------------------------------------------------------
# Initialize separate engines for polling set(s)
HiPRIOpoller = zmq.Poller()
LoPRIOpoller = zmq.Poller()
# ------------------------------------------------------------
# Associate
HiPRIOpoller.register( socket_0_pull, zmq.POLLIN ) # 0:
LoPRIOpoller.register( ... , zmq.POLLIN ) # 1:
LoPRIOpoller.register( ... , zmq.POLLIN ) # 2:
...
# ------------------------------------------------------------
# Detect, who is waiting in front of the closed door
aListOfHiPRIOevents = HiPRIOpoller.poll( timeout = 0.200 ) # 200 [us]
aListOfLoPRIOevents = LoPRIOpoller.poll( timeout = 0 ) # no wait at all
# ------------------------------------------------------------
# Now AFTER you have a COMPLETE view what is waiting there
# one
# CAN & SHALL ADAPT order / scope of event-handling,
# IMMUNE to infinite-PRIO-event-flow.
...
# ------------------------------------------------------------
Poller.poll() method returns a list of events that are ready to be processed. This is a list of tuples of the form ( socket, event ), where the first element is { a-0MQ-Socket-instance | integer-system-native-fd }, and the second is a poll-event mask ( POLLIN, POLLOUT ). It is common to call this decorated as aDictOfEVENTs = dict( aPoller.poll() ), which turns the list of tuples into a mapping of { aSocket : anEvent, ... } if one wishes to.
finally,
Epilogue: a Tribute to Margaret HAMILTON and her MIT team
If all the story was nothing else to inspire our thoughts, Margaret's brave efforts have taught us a lot about professional system design. May learn a lot from this truly pioneering epoch - more about what computer science has implemented on penny-scaled resources' footprints already in early 60-ies, that today systems' designs are ( if not worse ) quite too often in debt for ...
( almost ) Post-festum:
With a pleasure & delight, let me share the White-House news about a fair & honorable moment when Margaret HAMILTON has been recently awarded a Presidential Medal of Freedom.
I am personally very glad NASA folks have found a few words on excellence for such brilliant personalities Margaret Hamilton and her team-mates out of question are.
In this context one may consider an historical imperative to compare NASA remarks on ZERO-BUG ever found in Margaret Hamilton's team products, deployed --- in Apollo, Skylab, Space Shuttle and the first U.S. fly-by-wire aircraft --- with somehow other kind of experience gathered on products from one other medalist, awarded just "two rows" before Margaret.
An uncompromised excellence simply remains The Excellence.
My hat is raised, dear Mrs. Hamilton.
Let your enlightment shows us, the mortals, the better ways forwards.
来源：https://stackoverflow.com/questions/38633359/zeromq-how-to-prioritise-sockets-in-a-poll-method