python zeromq並行管道多個消費者（工人）

Question

一起嗨！

我想通過 zeromq 組合來自不同 python 程序的數據。 我認為對於這項工作，最好的解決方案是並行管道，如本例中“3.並行管道（並行管道模式）”中所述。

所以一開始我想用非常簡單的例子來測試這個功能。 因為我剛剛使用了您可以在示例中找到的三種模式producer 、 consumer和resultcollector 。 我只是做了一些小改動：

制片人

import zmq

context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.bind("tcp://*:5566")

work_message = "start working"
socket.send_json(work_message)

消費者_1

import random
import zmq


context = zmq.Context()
consumer_id = 199

#receiving work
consumer_receiver = context.socket(zmq.PULL)
consumer_receiver.connect("tcp://localhost:5566")

#forewarding results
consumer_sender = context.socket(zmq.PUSH)
consumer_sender.bind("tcp://*:5500")

msg = consumer_receiver.recv_json()

for i in (0, 100):
    if msg == "start working":
        data = "id: " + str(consumer_id) + "; Hello"
        consumer_sender.send_json(data)

結果收集器

#coding:utf-8

import zmq

context = zmq.Context()
result_receiver = context.socket(zmq.PULL)
result_receiver.connect("tcp://localhost:5599")

result = result_receiver.recv_json()

collected_data = []

for i in (0, 10000):
    collected_data.append(result)
    
print(collected_data)

所以他們之間的主要溝通是有效的。 但現在我嘗試添加另一個提供數據的消費者（工人）。 正如您在我鏈接的示例中所見，這應該是可能的。 在這里你可以看到我的第二個消費者（工人）：

消費者_2

import random
import zmq


context = zmq.Context()
consumer_id = 10

#receiving work
consumer_receiver = context.socket(zmq.PULL)
consumer_receiver.connect("tcp://localhost:5566")

#forewarding results
consumer_sender = context.socket(zmq.PUSH)
consumer_sender.bind("tcp://*:5599")

msg = consumer_receiver.recv_json()

for i in (0, 100):
    if msg == "start working":
        data = "id: " + str(consumer_id) + "; World"
        consumer_sender.send_json(data)

它與consumer_1 幾乎相同。 但是當我想同時運行兩個消費者時，我收到錯誤消息“ZMQError: Adress in use”：

---------------------------------------------------------------------------
ZMQError                                  Traceback (most recent call last)
<ipython-input-1-af8297fe1137> in <module>
     12 #forewarding results
     13 consumer_sender = context.socket(zmq.PUSH)
---> 14 consumer_sender.bind("tcp://*:5599")
     15 
     16 msg = consumer_receiver.recv_json()

~\Anaconda3\lib\site-packages\zmq\sugar\socket.py in bind(self, addr)
    171 
    172         """
--> 173         super().bind(addr)
    174         return self._bind_cm(addr)
    175 

zmq/backend/cython/socket.pyx in zmq.backend.cython.socket.Socket.bind()

~\Anaconda3\lib\site-packages\zmq\backend\cython\checkrc.pxd in zmq.backend.cython.checkrc._check_rc()

ZMQError: Address in use

我的錯在哪里？ 必須可以將不同的程序推送到單個端口（如示例端口 5599）還是我錯了？ 如果您認為這是一個愚蠢的問題，我對編程很陌生。

如果你們能在這里幫助我，那就太好了。

非常感謝！

Answer 1

問： “我的錯在哪里？……好吧明白了，但那我該怎么辦？”

A ：
最安全的方法（即使是 ZeroMQ 默認值也可以並且確實會隨着版本的變化而演變）是設計某種程度的自我保護穩健性。

我有過讓自己的代碼死鎖硬件的經歷，重啟是修復設計不當代碼的最后手段——所以如果你能從我的錯誤中吸取教訓，那就更好了。

雖然這些模板不是萬能的，但它們為進一步完善的自我保護、近實時的分布式計算提供了結構，即使在這些情況下，如果發生崩潰，也要保持規則拇指優雅地拆除並將所有阻塞的資源釋放回操作系統（這樣下次進程重新啟動時，它會發現它是“自己的” .bind() - 准備好端口釋放，准備重新bind()一個 ZeroMQ 訪問點回到同一個端口（如果沒有優雅地釋放並且zmq.LINGER在無限等待循環中保留一個Context()實例，如果沒有正確設置不這樣做 - 不要依賴“當前”版本隱式默認值，它可能會再次更改為其他一些默認值，並且任何基於此類的假設都將停止工作）。

collector的模板應如下所示：

import zmq

context = zmq.Context()
#------------.Context()-call error-handling
...
result_receiver = context.socket( zmq.PULL )
#------------------------.socket()-call error-handling
...
result_receiver.setsockopt( zmq.LINGER,  0 )    # a default value might 
change in future versions as it did already 
#--------------.setsockopt() settings for other attributes
...
result_receiver.bind( "tcp://localhost:5599" )  # a BIND()-side, many .connect() here
#--------------.bind() error-handling
...
collected_data = []
poll_will_not_block_longer_than_ms = 5
try:
    #------------------------------------- INTENDED PATH:
    for i in range( 0, 10000 ):
        if ( 0 == result_receiver.poll( poll_will_not_block_longer_than_ms ):
             #--------------.poll() has nothing to deliver on .recv()
             # do some maintenance work
             ...
             # do some "right-sized" sleep ( by a countdown residual et al )
        else:
             #--------------.poll() SIG'd a .recv()-able message:
             result = result_receiver.recv_json()
             #-----------------------.recv_json()-call error-handling
             ...
             collected_data.append( result )
             #-------------.append() can silently grow till RAM exhausted
             #                       can kill the O/S, be careful in production
     #---------------------------------------------------
except KeyboardInterrupt:
     #aKeyboardInterrupt Handler PATH:
     ...
except *other*Exception:
     #anOther*Exception Handler PATH:
     ...
except:
     #all remaining Exceptions' handler code:
     ...

finally:
     #----------------------------------------- CORE ZeroMQ TIDY-UP CODE:
     result_receiver.close()
     context.term()
     ...
#---------------------------------------------- all resources gracefully released

print( collected_data )

同樣， consumer轉發器的模板應如下所示：

import zmq

DOWNLINK_URL = "tcp://*:5566"
UPLINK_URL   = "tcp://*:5599"

context = zmq.Context()
#------------.Context()-call error-handling
...
consumer_receiver = context.socket( zmq.PULL )
#--------------------------.socket()-call error-handling
...
consumer_receiver.setsockopt( zmq.LINGER,  0 )    # a default value might 
change in future versions as it did already 
#----------------.setsockopt() settings for other attributes
...
consumer_receiver.connect( DOWNLINK_URL )
#----------------.connect()-call error-handling
...
consumer_sender = context.socket( zmq.PUSH )
#------------------------.socket()-call error-handling
...
consumer_sender.setsockopt( zmq.LINGER,  0 )    # a default value might 
change in future versions as it did already 
#--------------.setsockopt() settings for other attributes
...
consumer_sender.connect( UPLINK_URL )
#--------------.connect()-call error-handling
...
consumer_id = 199
poll_will_not_block_longer_than_ms = 5
try:
    #------------------------------------- INTENDED PATH:
    for i in range( 0, 10000 ):
        if ( 0 == consumer_receiver.poll( poll_will_not_block_longer_than_ms ):
             #---------------------.poll() has nothing to deliver on .recv()
             # do some maintenance work
             ...
             # do some "right-sized" sleep ( by a countdown residual et al )
        else:
             #---------------------.poll() SIG'd a .recv()-able message:
             result = consumer_receiver.recv_json()
             #-------------------------.recv_json()-call error-handling
             ...
             data = "id: " + str( consumer_id ) + "; Hello"
             consumer_sender.send_json( data, zmq.NOBLOCK )
             #--------------.send_json()-call error-handling
             ...
     #---------------------------------------------------
except KeyboardInterrupt:
     #aKeyboardInterrupt Handler PATH:
     ...
except *other*Exception:
     #anOther*Exception Handler PATH:
     ...
except:
     #all remaining Exceptions' handler code:
     ...

finally:
     #----------------------------------------- CORE ZeroMQ TIDY-UP CODE:
     for aSocket in ( consumer_receiver,
                      consumer_sender ):
         aSocket.close()
     
     context.term()
     ...
#---------------------------------------------- all resources gracefully released
...