如何使用一些过滤器使用来自 Kafka 主题的消息？

Question

I am writing a simple Kafka producer-consumer program in Java, where I have produced data like this:

• key:"a" value:"{25,223,465}"

• key:"a" value:"{26,323,56}"

• key:"a" value:"{62,256,652}"

• key:"a" value:"{26,227,42}"

• key:"b" value:"{4352,234,65342}"

• key:"b" value:"{243,22347,434}"

I am able to consume the messages using consumer.poll(10000) , But now I want to consume data something like how many a records and how many b records are present in the Kafka topic.

If I relate this to SQL

select count(*) from 'mytopic' where key='a'

select count(*) from 'mytopic' where key='b'

Please provide me code in java if possible

Answer 1

Kafka doesn't work in this way, a consumer just reads from the latest committed offset in a partition (or from the beginning) and sequentially all the messages. What you can do is filtering in your application. For your purpose, instead of writing an application using just consumer API you can use a Kafka Streams API based application that provides you a DSL for doing operating like mapping, filtering, ... really easily. More information here:

https://kafka.apache.org/documentation/streams/