Connecting to multiple clusters spring kafka

Question

I would like to consume messages from one kafka cluster and publish to another kafka cluster. Would like to know how to configure this using spring-kafka?

Answer 1

Simply configure the consumer and producer factories with different bootstrap.servers properties.

If you are using Spring Boot, see

https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#spring.kafka.consumer.bootstrap-servers

and

https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-application-properties.html#spring.kafka.producer.bootstrap-servers

If you are creating your own factory @Bean s, set the properties there.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#connecting

Answer 2

you can use spring cloud stream kafka binders.

create two stream one for consume and one for producing.

for consumer

public interface InputStreamExample{
    String INPUT = "consumer-in";
    @Input(INPUT)
    MessageChannel readFromKafka();
}

for producer

public interface ProducerStreamExample{

    String OUTPUT = "produce-out";

    @Output(OUTPUT)
    MessageChannel produceToKafka();
}

for consuming message use:

@StreamListener(value = ItemStream.INPUT)
public void processMessage(){
    /*
    code goes here
    */
}

for producing

//here producerStreamExample is instance of ProducerStreamExample 
producerStreamExample.produceToKafka().send(/*message goes here*/);

Now configure consumer and producer cluster using binder, you can use consumer-in for consumer cluster and produce-out for producing cluster.

properties file

spring.cloud.stream.binders.kafka-a.environment.spring.cloud.stream.kafka.binder.brokers:<consumer cluster>
#other properties for this binders

#bind kafka-a to consumer-in
spring.cloud.stream.bindings.consumer-in.binder=kafka-a  #kafka-a binding to consumer-in
#similary other properties of consumer-in, like
spring.cloud.stream.bindings.consumer-in.destination=<topic>
spring.cloud.stream.bindings.consumer-in.group=<consumer group>


#now configure cluster to produce 

spring.cloud.stream.binders.kafka-b.environment.spring.cloud.stream.kafka.binder.brokers:<cluster where to produce>


spring.cloud.stream.bindings.produce-out.binder=kafka-b    #here kafka-b, binding to produce-out
#similary you can do other configuration like topic
spring.cloud.stream.bindings.produce-out.destination=<topic>

Refer to this for more configuration: https://cloud.spring.io/spring-cloud-stream-binder-kafka/spring-cloud-stream-binder-kafka.html