使用接收器连接器将Kafka转换为Google Pub / Sub

Question

Sorry for the basic question.

Requirement: Need to send json payload from spring boot application to google pub sub.

I have started implementing kafka producer and publishing message to topics and from there to kafka consumer is receiving the json.

Now i need to send the json payload from kafka consumer to google pub/sub. Iam confused, should i need kafka consumer for this.. Or just from kafka producer to google pub sub i can send by using the kafka connector.jar and configuring in the topics in the properties. Any help is much appreciated for this implementation

Answer 1

If you want to get messages from Kafka into Google Cloud Pub/Sub, you should not need to write your own consumer. You can create an instantiation of the Google Cloud Pub/Sub Kafka connector . The Kafka Connect service is usually part of the Kafka deployment itself, so you just need to start an instance of it configured to run the Cloud Pub/Sub sync connector. The README file for the connector details the steps that need to be taken, but to summarize:

1. Download the latest release and build it via the mvn package command.
2. Copy target/cps-kafka-connector.jar to a place in your Java path so it accessible when Kafka Connect runs.
3. Copy the sink config and change it to point to the appropriate Cloud Pub/Sub project and topic and the appropriate Kafka topics.
4. Make a copy of the config/connect-standalone.properties or config/connect-distributed.properties based on whether or not you want to have a single instance or multiple instances of the connector running.
5. Update the key.converter and value.converter properties to be org.apache.kafka.connect.storage.StringConverter . This way, the connector will not try to interpret the data being passed but will instead just pass the JSON straight through to Cloud Pub/Sub.
6. Start up the connector with the appropriate command based on the standalone vs. distributed connector, eg, bin/connect-standalone.sh <standalone config file> <connector config file> .

Messages should now be flowing from Kafka into Google Cloud Pub/Sub. If you are just using Kafka to go from Spring Boot to Cloud Pub/Sub, then you could avoid the Kafka step by setting up an outbound channel adapter to send messages to Cloud Pub/Sub directly .