Deserialisation to Protobuf object in Python

Question

I have passed a Protobuf object in a Kafka producer and am receiving a byte array on the consumer side. Now I want to deserialize that response again back to a Protobuf object, but I am unable to do that. How can I do that?

Here is my consumer:

from email import message
from kafka import KafkaConsumer
import json
from simple import simple_message
import check_pb2

consumer = KafkaConsumer('latest', api_version=(0, 10, 1),
                         group_id='my-group', enable_auto_commit=False,
                         bootstrap_servers=['localhost:9092'])#,
           #value_deserializer=lambda m: json.loads(m.decode('ascii')))
for message in consumer:
    print(message)

I tried to use Confluent but getting

Kafka Exception : {} KafkaError{code=_VALUE_DESERIALIZATION,val=-159,str="Unknown magic byte. This message was not produced with a Confluent Schema Registry serializer"

Answer 1

If you're not using Confluent Schema Registry, use ParseFromString .

for message in consumer:
    value = ParseFromString(message.value)  # example 
    print(value)

If you are using the Schema Registry, then Confluent has a Protobuf deserializer class Example - https://github.com/confluentinc/confluent-kafka-python/blob/master/examples/protobuf_consumer.py