Kafka Connect or Kafka Streams?

Question

I have a requirement to read messages from a topic, enrich the message based on provided configuration (data required for enrichment is sourced from external systems), and publish the enriched message to an output topic. Messages on both source and output topics should be Avro format.

Is this a good use case for a custom Kafka Connector or should I use Kafka Streams?

Why I am considering Kafka Connect?

• Lightweight in terms of code and deployment
• Configuration driven
• Connection and error handling
• Scalability
• I like the plugin based approach in Connect. If there is a new type of message that needs to be handled I just deploy a new connector without having to deploy a full scale Java app.

Why I am not sure this is good candidate for Kafka Connect?

• Calls to external system
• Can Kafka be both source and sink for a connector?
• Can we use Avro schemas in connectors?
• Performance under load
• Cannot do stateful processing (currently there is no requirement)
• I have experience with Kafka Streams but not with Connect

Answer 1

Use both?

Use Kafka Connect to source external database into a topic.

Use Kafka Streams to build that topic into a stream/table that can then be manipulated.

Use Kafka Connect to sink back into a database, or other system other than Kafka, as necessary.

---

Kafka Streams can also be config driven, use plugins (ie reflection), is just as scalable, and has no different connection modes (to Kafka). Performance should be the similar. Error handling is really the only complex part. ksqlDB is entirely "config driven" via SQL statements, and can connect to external Connect clusters, or embed its own.

Avro works for both, yes.

Some connectors are temporarily stateful, as they build in-memory batches, such as S3 or JDBC sink connectors