Processing Complex Avro messages using Kafka Streams

Question

I am working on a POC on Kafka Streams in which I am processing avro messages using Kafka Streams. Thing is that my Avro message has mix of simple and complex types, so I am finding it difficult in processing it.

My Avro Schema looks like below.

{"type":"record",
"namespace": "com.test",
"name": "backoffice",
"fields": [ {"name": "accountid","type": "string"},
{"name":"amendmentpositionid","type": "int"},
{"name":"booking","type":
{"type":"array","items":
{"namespace":"com.saxo",
"name":"bookingfields",
"type":"record",
"fields":
[{"name":"accountid","type":"string"},{"name":"clientid","type":"int"},
{"name":"clientname","type":"string"},{"name":"exerciseid","type":"int"},
{"name":"hedgeid","type":"int"},{"name":"originatingpositionid","type":"int"},
{"name":"positionid","type":"int"},{"name":"relatedpositionid","type":"int"} ]}}}]}

Input Data is mentioned as below

{"accountid":"1234","amendmentpositionid":1234,"booking":[{"accountid":"898","clientid":333,"clientname":"Non ","exerciseid":2,"hedgeid":100

I need it to flatten and look like as mentioned below before storing it to database.

1234,1234,898,333,NON,2,100

To achieve this I am trying to use Kafka Streams flatmapvalues operation but somehow I am not able to retain id and date in my final output.

My kafka Streams application looks like below.

package com.test.office.KafkaStreams;

import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig;
import io.confluent.kafka.serializers.KafkaAvroSerializerConfig;
import io.confluent.kafka.streams.serdes.avro.GenericAvroSerde;
import org.apache.avro.generic.GenericData;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
//import io.confluent.kafka;
//import org.apache.kafka.common.serialization.Serdes.
//import io.confluent.kafka.serialiszers.AbstractKafkaAvroSerDeConfig;
import io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KStreamBuilder;
import com.saxo.backoffice;
import io.confluent.kafka.streams.serdes.avro.GenericAvroSerde;
import org.apache.kafka.streams.kstream.KeyValueMapper;

import javax.swing.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Properties;

public class KafkaAvroSchemaRegistry {


    public static void main(String[] args) {

        Properties properties = new Properties();
        properties.put(StreamsConfig.APPLICATION_ID_CONFIG, "Kafka Avro Topic 8");
        properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "server1");
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, SpecificAvroSerde.class);
        properties.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://server2:8081");

        KStreamBuilder builder = new KStreamBuilder();
        KStream<String, testSpecific> testspecific1 = builder.stream("topic10");

        KStream<String,testspecific> output1 = testspecific1.peek((key,value) -> System.out.println(key + value.toString()));


output1.print();
KStream<String,String> test = testspecific1.flatMapValues(value -> value.Booking());

test.print()
        KafkaStreams streams = new KafkaStreams(builder, properties);

        streams.cleanUp();
        streams.start();


        // print the topology
       // System.out.println(streams.toString());

        // shutdown hook to correctly close the streams application
        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
    }
}

Can someone point me in the right direction?

Answer 1

somehow I am not able to retain id and date in my final output.

Because you're only mapping out into getMessage() .

Try getting all the fields

flatMapValues(value -> 
    String.format("%d,%s,%s", value.getId(), value.getDate(), value.getMessage());