Unsupported source data type: STRUCT error in JDBC Postgres Sink Connector when consuming from Kafka topic

Question

I am receiving this error when trying to connect a JDBC Connector to a topic in Kafka and I have not been able to fix it. The data can successfully be serialized and deserialized using the schema and it is validated before sending it into a topic. Also, a basic consumer on the topic can receive the messages sent into the topic but the JDBC connector cannot map the values in the message to the table. I have tried with both auto.create = true and auto.evolve = true with no luck. For what it's worth, I have a very similar table,schema, and JDBC connector for a different topic and it can successfully insert values in the database and it also has an auto-increment key called id that doesn't need to be specified in the message for it to work (just in case your first thought is the missing id field in the Avro schema).

This is the error I get:

Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. Error: Unsupported source data type: STRUCT (org.apache.kafka.connect.runtime.WorkerSinkTask)

org.apache.kafka.connect.errors.ConnectException: Unsupported source data type: STRUCT

at io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.bindField(GenericDatabaseDialect.java:1627)

at io.confluent.connect.jdbc.dialect.DatabaseDialect.bindField(DatabaseDialect.java:608)

at io.confluent.connect.jdbc.sink.PreparedStatementBinder.bindField(PreparedStatementBinder.java:186)

at io.confluent.connect.jdbc.sink.PreparedStatementBinder.bindKeyFields(PreparedStatementBinder.java:154)

at io.confluent.connect.jdbc.sink.PreparedStatementBinder.bindRecord(PreparedStatementBinder.java:102)

at io.confluent.connect.jdbc.sink.BufferedRecords.flush(BufferedRecords.java:184)

at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:80)

at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:84)

at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:601)

at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:333)

at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:234)

at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:203)

at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)

at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

[2022-03-17 00:55:12,913] ERROR WorkerSinkTask{id=sink_postgres_filing-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)

org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.

at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:631)

at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:333)

at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:234)

at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:203)

at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)

at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:243)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: org.apache.kafka.connect.errors.ConnectException: Unsupported source data type: STRUCT

at io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.bindField(GenericDatabaseDialect.java:1627)

at io.confluent.connect.jdbc.dialect.DatabaseDialect.bindField(DatabaseDialect.java:608)

at io.confluent.connect.jdbc.sink.PreparedStatementBinder.bindField(PreparedStatementBinder.java:186)

at io.confluent.connect.jdbc.sink.PreparedStatementBinder.bindKeyFields(PreparedStatementBinder.java:154)

at io.confluent.connect.jdbc.sink.PreparedStatementBinder.bindRecord(PreparedStatementBinder.java:102)

at io.confluent.connect.jdbc.sink.BufferedRecords.flush(BufferedRecords.java:184)

at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:80)

at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:84)

at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:601)

... 10 more

When setting auto.create=true and auto.evolve=true the error message becomes more specific and says io.confluent.connect.avro.Union (STRUCT) type doesn't have a mapping to the SQL database column type .

This is my JDBC sink connector settings:

curl -i -X PUT http://localhost:8083/connectors/sink_postgres_filing/config \
     -H "Content-Type: application/json" \
     -d '{
            "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
            "value.converter.schema.registry.url": "http://schema-registry:8081",
            "value.converter.schema.enable": true,
            "value.converter": "io.confluent.connect.avro.AvroConverter",
            "key.converter.schema.registry.url": "http://schema-registry:8081",
            "key.converter": "org.apache.kafka.connect.storage.StringConverter",
            "connection.url": "jdbc:postgresql://host.docker.internal:5432/filing-api",
            "connection.user": "postgres",
            "connection.password": "12345",
            "insert.mode": "upsert",
            "pk.mode": "record_value",
            "pk.fields": "filing_number",
            "topics": "filing",
            "errors.log.enable":true,
            "errors.log.include.messages":true
         }'

This is my Avro schema:

curl -i -X POST http://localhost:8081/subjects/filing-value/versions/ \
      -H "Content-Type: application/vnd.schemaregistry.v1+json" \
      -d '{
             "schema": "{\"namespace\":\"io.Filing\",\"type\":\"record\",\"name\":\"Filing\",\"fields\":[{\"name\":\"filing_number\",\"type\":[\"long\"]},{\"name\":\"field1\",\"type\":[\"string\", \"null\"]},{\"name\":\"field2\",\"type\":[\"string\", \"null\"]},{\"name\":\"field3\",\"type\":[\"string\", \"null\"]},{\"name\":\"field4\",\"type\":[\"string\", \"null\"]},{\"name\":\"field5\",\"type\":[\"long\", \"null\"]},{\"name\":\"field6\",\"type\":[\"long\", \"null\"]},{\"name\":\"field7\",\"type\":[\"string\", \"null\"]},{\"name\":\"field8\",\"type\":\"string\"}]}",
             "schemaType": "AVRO"
          }'

This is my PostgreSQL table:

CREATE TABLE IF NOT EXISTS filing
(
    id bigserial PRIMARY KEY,
    filing_number bigint NOT NULL UNIQUE,
    field1 character varying,
    field2 character varying,
    field3 character varying,
    field4 character varying,
    field5 bigint,
    field6 bigint,
    field7 character varying,
    field8 character varying NOT NULL
);

Answer 1

I have solved the issue. But to be quite honest, I am not sure what I did. I changed the jdbc connector settings to auto.create = true and auto.evolve = true , insert.mode = insert , pk.mode = none , and I removed the pk.fields attribute.

Essentially, I wanted the connector to dump it into the database in its own way without my interference to see if it would work. Once I did this, I was able to change the settings back one by one and it worked.

I had produced some messages before of a different format to the topic so perhaps those were the messages that were causing the issue but I'm not entirely sure because I also deleted the docker container hosting my kafka broker to clear the messages prior before changing the settings to those I mentioned and I also set error.tolerance = "all" and they didn't seem to work.