Simple client/server ZMQ in Python to send multiple lines per request

Question

This is my first exposure to ZMQ under Python and I want the server to sent multiple lines when it receives a request from the client. The code that I added to the example offered by ZMQ on the server side is:

with open("test.txt", 'r') as f:
    for line in f:
        socket.send_string(line.rstrip("\n"))

The question is how to make the server send all lines or how to make the client not to send a request before server finishes sending all the lines from test.txt

Client

import zmq
context = zmq.Context()
print("Connecting to hello world server")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
for request in range(10):
    print("Sending request %s" % request)
    socket.send(b"Hello")
    message = socket.recv()
    print("Received reply %s [ %s ]" % (request, message))

Server

import time
import zmq

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")

while True:
    #  Wait for next request from client
    message = socket.recv()
    print("Received request: %s" % message)

    #  Do some 'work'
    time.sleep(1)

    #  Send reply back to client
    with open("test.txt", 'r') as f:
        for line in f:
            socket.send_string(line.rstrip("\n"))

Client log

Connecting to hello wolrd server
Sending request 0
Received reply 0 [ This is test line 1 ]
Sending request 1

This is where it stops, as the server generated the error shown bellow:

Server log

line 324, in send_string
     return self.send(u.encode(encoding), flags=flags, copy=copy)
File "socket.pyx", line 571, in zmq.backend.cython.socket.Socket.send (zmq/backend/cython/socket.c:5319)
File "socket.pyx", line 618, in zmq.backend.cython.socket.Socket.send (zmq/backend/cython/socket.c:5086)
File "socket.pyx", line 181, in zmq.backend.cython.socket._send_copy (zmq/backend/cython/socket.c:2081)
File "checkrc.pxd", line 21, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:6032)
zmq.error.ZMQError: Operation cannot be accomplished in current state

Process finished with exit code 1 

test.txt

This is test line 1
This is test line 2
This is test line 3
This is test line 4
This is test line 5

Answer 1

Here is the solution that I came up with...just in case someday, someone might need help with a similar situation. The idea is to packetize all lines in one single message and send it back to client. It seems that the it works is that for every request made by client , the server needs to make a reply and only one reply. At least that's how I see it..

Replace code on the server side

#  Send reply back to client
with open("test.txt", 'r') as f:
    for line in f:
        socket.send_string(line.rstrip("\n"))

with:

#  Send reply back to client
with open("test.txt", 'r') as f:
    message = '%s' % f.readlines()
    print(message)
    print(type(message))
    socket.send_string(message)

Client request

Connecting to hello world server
Sending request 0
Received reply 0 [ ['This is test line 1\n', 'This is test line 2\n', 'This is test line 3\n', 'This is test line 4\n', 'This is test line 5\n', 'This is test line 6\n', 'This is test line 7\n', 'This is test line 8\n', 'This is test line 9\n', 'This is test line 10\n', '\n'] ]
Sending request 1
Received reply 1 [ ['This is test line 1\n', 'This is test line 2\n', 'This is test line 3\n', 'This is test line 4\n', 'This is test line 5\n', 'This is test line 6\n', 'This is test line 7\n', 'This is test line 8\n', 'This is test line 9\n', 'This is test line 10\n', '\n'] ]
 ....
 ....
 and so on up to 10 requests

Server response

Received request: Hello
['This is test line 1\n', 'This is test line 2\n', 'This is test line 3\n', 'This is test line 4\n', 'This is test line 5\n', 'This is test line 6\n', 'This is test line 7\n', 'This is test line 8\n', 'This is test line 9\n', 'This is test line 10\n', '\n']
<type 'str'>
Received request: Hello
['This is test line 1\n', 'This is test line 2\n', 'This is test line 3\n', 'This is test line 4\n', 'This is test line 5\n', 'This is test line 6\n', 'This is test line 7\n', 'This is test line 8\n', 'This is test line 9\n', 'This is test line 10\n', '\n']
<type 'str'>
....
....
....
and so on....

Now that this is solved, the next question would be: what kind of request client needs to send to be able to accept responses from the server in line-by-line fashion. I will update the response if I will have a solution or you can feel free to participate with your own.

Answer 2

Well, you came up with my preferred solution, that is, to just send the whole thing as a single message, using separate frames if necessary. That said, the reason it will allow you to only send a single reply is because you're using a REQ-REP socket pair and when using such a pair you must follow a simple "request-reply-request-reply" pattern. Each communication must start with one request, and the next message must be one reply, and the next one request, etc.

To work around this you have several options:

• As you did, package up everything you want to send in a single message. In your case, so long as there are not constraints that prevent it, that would be the preferred option because all of the data is actually one logical message, coming from one logical file, and it's all going to be composed into a single logical piece of data on the receiving end, I assume.
• If you want to keep the REQ-REP socket pair, then you can send a request, followed by a reply, and then your next request can just be "MORE" or something like that, followed by the next piece of data, and just keep on requesting "MORE" until your client replies with something like "DONE", at which point you know you have all of the data and can quit requesting more.
• Or, if you prefer to be able to do multiple replies to a single request, or request multiple times before receiving a reply, then you'll want to use a ROUTER-DEALER socket pair instead of REQ-REP . In this case, your DEALER socket would take the place of your REQ socket, and ROUTER would take the place of REP . Check out the manual and ask a new question if you get stuck on how to implement ROUTER-DEALER .