Kafka spring boot 应用程序生产者，无法使用 Avro 格式的 Kafka Sink Connector 反映这一点

Question

My target is I have spring boot application kafka producer with Avro serializing property and i am excepting the message which is pushed to respective topic that should access by confluent Sink Connector and insert into mysql/Oracle database tables, am able to produce Avro serialize and spring boot consumer can Avro deserialize but my Sink connector is not working, I am not able to catch what kind of payload sink connector is excepting and how should be spring boot producer coded to push message such a way that sink connector can cop-up with that properties

Thanks in advance:)

This is application.yml in spring boot application

server: 
    port: 9000
    spring.kafka:
    bootstrap-servers: "localhost:9092"
    properties:
      schema.registry.url: "http://localhost:8081"
      specific.avro.reader: true
    producer:
      key-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
      value-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
    app:
      topic: event_pay2

This is payload for creation of schema from spring boot application

{
  "schema": {
    "type": "struct",
    "fields": [
      {
        "type": "string",
        "optional": false,
        "field": "userid"
      },
      {
        "type": "string",
        "optional": false,
        "field": "regionid"
      },
      {
        "type": "string",
        "optional": false,
        "field": "gender"
      }
    ],
    "optional": false,
    "name": "oracle.user"
  },
  "payload": {
    "userid": "User_1",
    "regionid": "Region_5",
    "gender": "FEMALE"
  }
}

Pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.0.5.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.kafka</groupId>
<artifactId>kafka-producer-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafka-producer-example</name>
<description>Demo project for Spring Boot</description>

 <repositories>
    <repository>
        <id>confluent</id>
        <url>http://packages.confluent.io/maven</url>
    </repository>
</repositories>

<properties>
    <java.version>1.8</java.version>
    <confluent.version>4.0.0</confluent.version>
</properties>

<dependencies>
<dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <scope>test</scope>
    </dependency>
     <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-avro-serializer</artifactId>
        <version>${confluent.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.8.2</version>
</dependency>
</dependencies>

<build>
 <pluginManagement>
    <plugins>
    <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.8.2</version>
        <executions>
            <execution>
                <phase>generate-sources</phase>
                <goals>
                    <goal>schema</goal>
                </goals>
                <configuration>
                    <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
                    <outputDirectory>${project.build.directory}/generated/avro</outputDirectory>
                </configuration>
            </execution>
        </executions>
    </plugin>


        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
    </plugins>
     </pluginManagement>
</build>

</project>

This is my rest call how am pushing messgae into kafka topic

@PostMapping("/publish/avrodata")
public String sendMessage(@RequestBody String request) {
    sender.send(request);
    return "Published successfully";
}

Finally My sink connector

"name": "JDBC_Sink_EVENT_PAY",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "event_pay2",
"connection.url": "jdbc:mysql://localhost:3306/user",
"connection.user": "****",
"connection.password": "****",
"auto.create": "true",
"auto.evolve":"true",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://localhost:8081",
"key.converter.schemas.enable": "true",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8081",
 "value.converter.schemas.enable": "true"

Answer 1

Always, always debug your topic before setting up a Connector. Use kafka-avro-console-consumer for this. If that doesn't work, then Connect + AvroConverter likely won't work either (in my experience), and you can reduce the problem space.

---

If I read your code correctly, you've sent a String, and not an Avro object

public String sendMessage(@RequestBody String request) {
    sender.send(request);  // <--- here and ^^^^^^ here
    return "Published successfully";
}

Instead, you need to parse your input request into an object that was created as part of your /src/main/resources/avro schema data, not just forward through the incoming request as a string.

And that AVSC file might look something like

{
  "type": "Record",
  "namespace": "oracle.user",
  "name": "User",
  "fields": [
      { "type": "string", "name: "user_id" },
      { "type": "string", "name": "regionid" },
      { "type" "string", "name" "gender" }
   ]
}

Which would create a oracle.user.User object, making your KafkaTemplate need to be something like KafkaTemplate<String, oracle.user.User> sender