每个 kafka 主题的火花流不同值解码器

Question

I need to create a Spark streaming that reads from several topic, and uses a different decoder per each topic (each topic contains a different avro-encoded obect):

def decode_avro(message):
    schem = avro.schema.parse(open("error_list.avsc").read())
    bytes_reader = io.BytesIO(message)
    decoder = avro.io.BinaryDecoder(bytes_reader)
    reader = avro.io.DatumReader(schem)
    return reader.read(decoder)

ssc = StreamingContext(sc, 2)
kvs = KafkaUtils.createDirectStream(ssc, [topic, topic2], {
    "metadata.broker.list": brokers}, valueDecoder = decode_avro)

I wan't to know if it is possible to specify different decoder callbacks per topic, or if it is possible to know the topic name on the decoder function (on this way I could used the topic name for the avro schema file and decode all messages in the same function)

Thank you

Answer 1

We also have this case where we read from different topics, with different message formats, and then process each topic and store the output into dedicated storage per source topic. The right way to go here, is to create multiple streams. Stream per topic, in the same application, with the same Spark context. Each stream will get the relevant ValueDecoder, and you may still read from multiple topics if they share the same format.