Kafka-Connect JDBC Sink reports null id during upsert

Question

I'm new to Kafka / Kafka Connect and I'm running into an issue with the confluent JDBC connector. Currently I am making use of the Confluent Community docker compose.

I can successfully create a source reading from a mysql DB into kafka.

curl -X POST \
-H "Content-Type: application/json" \
--data '{ "name": "college_mysql_source", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": 1, "connection.url": "jdbc:mysql://host.docker.internal:3306/...", "mode": "timestamp+incrementing", "timestamp.column.name": "updated_on", "topic.prefix": "college_mysql_", "poll.interval.ms": 1000, "table.whitelist": "college" } }' \
http://localhost:8083/connectors

The data enters Kafka as expected, each column correctly represented in Avro. If I create a consumer via CLI I can see the data is correct.

{
    "id":112525,
    "pim_id":{"long":78806},
    "college_name":{"string":"Western University of Health Sciences"},
    ...
}

If I create a simple JDBC sink to put the data into another mysql DB, things are totally fine:

curl -X POST -H "Content-Type: application/json" \
  --data '{"name": "weighted_average_mysql_sink_college", "config": {"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max":"1", "topics":"college_mysql_college", "connection.url": "jdbc:mysql://host.docker.internal:3306/...", "auto.create": "true", "insert.mode": "insert"}}' \
  http://localhost:8083/connectors

We correctly create a table and new records go in just fine with all fields (id included) populating correctly. However, if I instead create a sink that uses insert mode upsert I start to get errors.

curl -X POST -H "Content-Type: application/json" \
  --data '{"name": "weighted_average_mysql_sink_college", "config": {"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max":"1", "topics":"college_mysql_college", "connection.url": "jdbc:mysql://host.docker.internal:3306/...", "auto.create": "true", "insert.mode": "upsert", "pk.mode": "record_key", "pk.fields": "id"}}' \
  http://localhost:8083/connectors

This does correctly create the table and correctly establishes id as the primary key, so far so good, but now whenever it reads from the topic we get an error:

java.sql.BatchUpdateException: Column 'id' cannot be null

This is where I'm stuck. The data in the topic correctly has an ID field, and that ID field is used in the ID column if I'm not declaring the column as the PK. I've tried defining the table myself instead of allowing the sink to create the table, figuring maybe the table creation had some odd issue, but that doesn't seem to have been the case I get the exact same error either way. Any advice or direction on this would be appreciated, I'm hoping the solution is simple and I'm just missing something obvious to those with more experience will be able to point out to me.

Thanks!

Answer 1

您需要设置“pk.mode”：“record_value”