如何使用kafka connect jdbc sink将kafka主题连接到postgres数据库

Question

got errors while trying to connect kafka topics to postgres using jdbcsink connector

these are the error logs(see image) that i got when tried with the configuration

{
    "name": "temperature_jdbcsink",

    "config" : {

    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "task.max": "1",
    "topics": "temperature",

    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter": "io.confluent.connect.avro.AvroConverter",   
    "key.converter.schema.registry.url": "http://localhost:8081", 
    "value.converter.schema.registry.url": "http://localhost:8081",
    
    "transforms": "Flatten, RenameFields",
    "transfores.Flatten.type":"org.apache.kafka.connect.transforms.Flatten$value",
    "transforms.Flatten.deliniter":"_",
    
    "transforms.RenameFields.type": "org.apache.kafka.connect.transforms.ReplaceField$value",
    "transforms.RenameFields.renames": "value:value,timestamp:timestamp",
    
    "connection.url": "jdbc:postgresql://localhost:5432/jdbcsink",
    "connection.user": "postgres",
    "connection.password": "postgres",
    "insert.mode": "upsert",
    "batch.size":"2",
    "table.name.format": "temperature",
    "pk.mode":"none",
    "db.timezone": "Asia/Kolkata"
    }
}

https://i.stack.imgur.com/fXqO3.png
https://i.stack.imgur.com/fXqO3.png
https://i.stack.imgur.com/V5Btk.png

Answer 1

Error says Unknown magic byte . This seems there is data in the topic that wasn't produced using Confluent Avro serializer

For example, are your keys really Avro? This isn't common for storing data in JDBC sink since database primary keys are typically plain string or integer types. Therefore, use the respective converter for those, not Avro