[英]How to use python-trio with google protocol buffer?
I am trying to read some data streams using protobuf in python, and i want to use trio to make the client for reading the streams. 我试图在python中使用protobuf读取一些数据流,我想使用trio来使客户端读取流。 The protobuf has some method calls, and i find they do not work when i use trio streams. protobuf有一些方法调用,我发现当我使用三重流时它们不起作用。
Python client on a linux machine. Linux机器上的Python客户端。
import DTCProtocol_pb2 as Dtc
async def parent(addr, encoding, heartbeat_interval):
print(f"parent: connecting to 127.0.0.1:{addr[1]}")
client_stream = await trio.open_tcp_stream(addr[0], addr[1])
# encoding request
print("parent: spawing encoding request ...")
enc_req = create_enc_req(encoding) # construct encoding request
await send_message(enc_req, Dtc.ENCODING_REQUEST,client_stream, 'encoding request') # send encoding request
log.debug('get_reponse: started')
response = await client_stream.receive_some(1024)
m_size = struct.unpack_from('<H', response[:2]) # the size of message
m_type = struct.unpack_from('<H', response[2:4]) # the type of the message
m_body = response[4:]
m_resp = Dtc.EncodingResponse()
m_body
would be some bytes data, which I dont know how to decode. m_body
将是一些字节数据,我不知道如何解码。 Dtc.EncodingResponse()
is the protobuf method which would give a Dtc object which contains the response in a readable format. Dtc.EncodingResponse()
是protobuf方法,它将给出一个Dtc对象,该对象包含可读格式的响应。 (Dtc is the protobuf file). (Dtc是protobuf文件)。 But I get nothing here. 但我在这里什么都没得到。 When I did this script without trio, Dtc.EncodingResponse()
would give the full response in readable format. 当我在没有三重奏的情况下执行此脚本时, Dtc.EncodingResponse()
将以可读格式提供完整响应。
I am guessing the problem is that the "client_stream" is a trio stream object that only reads bytes, and so I probably need to use a ReceiveChannel
object instead. 我猜测问题是“client_stream”是一个只读取字节的三重流对象,因此我可能需要使用ReceiveChannel
对象。 But if this is true, I dont know how to do this. 但如果这是真的,我不知道该怎么做。
UPDATE: The answer below by Nathaniel J. Smith solves my problem. 更新:Nathaniel J. Smith在下面的答案解决了我的问题。
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_body)
I feel so silly, but I did not ParseFromString the data previously, and that was all it took. 我觉得很傻,但我之前没有ParseFromString数据,而这就是它所需要的。 Extremely grateful to all who gave replies. 非常感谢所有回复的人。 Hope this helps someone out there. 希望这有助于那里的人。
Like @shmee said in the comment, I think your code got mangled some by the edits... you should double-check. 就像@shmee在评论中所说的那样,我认为你的代码被编辑错了一些......你应该仔细检查一下。
When I did this script without trio,
Dtc.EncodingResponse()
would give the full response in readable format 当我在没有三重奏的情况下执行此脚本时,Dtc.EncodingResponse()
将以可读格式提供完整响应
I think you might have dropped a line when switching to Trio? 我想你在切换到Trio时可能会掉线? Dtc.EncodingResponse()
just creates a new empty EncodingResponse
object. Dtc.EncodingResponse()
只是创建一个新的空EncodingResponse
对象。 If you want to parse the data from m_body
into your new object, you have to do that explicitly, with something like: 如果要将m_body
的数据解析为新对象,则必须明确地执行此操作,例如:
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_body)
However, there's another problem... the reason it's called receive_some
is that it receives some bytes, but might not receive all the bytes you asked for. 但是,还有另一个问题......它被称为receive_some
的原因是它接收了一些字节,但可能没有收到你要求的所有字节。 Your code is assuming that a single call to receive_some
will fetch all the bytes in the response, and that might be true when you're doing simple test, but in general it's not guaranteed. 您的代码假设对receive_some
的单个receive_some
将获取响应中的所有字节,当您进行简单测试时,这可能是正确的,但通常不保证。 If you don't get enough data on the first call to receive_some
, you might need to keep calling it repeatedly until you get all the data. 如果在第一次调用receive_some
没有获得足够的数据,则可能需要反复调用它,直到获得所有数据。
This is actually very standard... sockets work the same way. 这实际上非常标准......套接字的工作方式相同。 That's why the first thing your server is sending an m_size
field at the beginning – it's so you can tell whether you've gotten all the data or not! 这就是为什么服务器在开始时发送m_size
字段的第一件事 - 这样你就可以知道你是否已经获得了所有数据!
Unfortunately, as of June 2019, Trio doesn't provide a helper to do this loop for you – you can track progress on that in this issue . 不幸的是,截至2019年6月,Trio没有为您提供帮助 - 您可以在此问题中跟踪进度。 In the mean time, it's possible to write your own. 与此同时,您可以自己编写。 I think something like this should work: 我认为这样的事情应该有效:
async def receive_exactly(stream, count):
buf = bytearray()
while len(buf) < count:
new_data = await stream.receive_some(count - len(buf))
if not new_data:
raise RuntimeError("other side closed the connection unexpectedly")
buf += new data
return buf
async def receive_encoding_response(stream):
header = await receive_exactly(stream, 4)
(m_size, m_type) = struct.unpack('<HH', header)
m_body = await receive_exactly(stream, m_size)
m_resp = Dtc.EncodingResponse()
m_resp.ParseFromString(m_size)
return m_resp
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.