Kafka Producer Guidelines java developers

Question

I am a beginner of kafka implementation. Can you give me some guidelines for writing a kafka producer to implement below requirements? I am setting up kafka topics with avro schema.

1. I need to publish complete data of my application to kafka topics when my app deployed into production/test env.
2. Next day onwards any modification on data (eg any attribute data has modified/NA/null) that should be published to kafka. Question: Should i send only modified data? Or should i send complete data with modified data? Could you tell me Which is good?

3.if any consumer joined to subscribe my topics very late, may be after 8 months after my application deployed in prod, he should be able to consume all data whatever before published.Is this possible to maintain more months in topics? If so how do we do that to inline with first two requirements.

Please Suggest some ideas really appreciated. Thanks

Currently i am writing a poc to brush up skills a sample producer which publish messages to kafka topic for any update on data. I am expecting ideal case for pushing messages to kafka topics

Answer 1

1. For publishing events to Kafka using Avro schema you can have a look at this article. Minimum producer config should be like this:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, io.confluent.kafka.serializers.KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://localhost:8081");
KafkaProducer producer = new KafkaProducer(props);

2. In my opinion it is better to publish always entire event to Kafka. Considering fact that you are using Avro you don't need to care about handling multiple null fields in your scheme or storing different schemas for same topic. Or if consumer will decide to consume from specific offset it will always has access to latest version of the event.

3. You can keep record in Kafka unlimited amount of time if you set retention.ms -1