I work in an organization where we must use the shared Kafka cluster. Due to internal company policy, the account we use for authentication has only t ...
I work in an organization where we must use the shared Kafka cluster. Due to internal company policy, the account we use for authentication has only t ...
First time working with Kafka and I've run into a problem. I have a following implementation of my consumer: Where the credentials should be imple ...
I have a job which is running with old flink Kafka consumer ( FlinkKafkaConsumer ) Now I want to migrate it to KafkaSource . But I am not sure what wi ...
Basically, that's the question: is there a way to get committed offsets and Kafka's offsets for each partition if using brod group subscriber? I'm us ...
I'm using Debezium to log changes in my database, and Debezium generates change events in a topic for each table that exists in my database.These chan ...
Let's begin with understanding(cmiiw), A topic in kafka divided to partition each partition is consumed by clients in the same group. Which client con ...
We have spring Kafka application with a customer deserializer class and we are using @KafkaListener annotation to receive messages. We noticed that we ...
I have an application with the need to pass messages into multiple layers of processing. I need to do this because all the new messages should be put ...
Use case There is an upstream system producing the data batches through Kafka (one batch has 10 millions of records on average). Start offset and exp ...
I am new to Kafka and would like to seek advice on what is the best practice to handle such scenario. Scenario: I have a spring boot application tha ...
If we have > 1 partition for a topic, then we can have a consumer group, now the different consumers in this group will split the partitions they w ...
I have a topic with two partitions,i'm using @RetryableTopic and i can see logs on my app console that INFO o.s.k.r.RetryTopicConfigurer - Received me ...
The problem is that we have a consumer reading from a queue and that consumer doing some updates to the database. Now we need to run multiple instance ...
I need to make Kafka consumers process all the messages with the same ID in each partition at once. For example, consider one topic containing all ord ...
I'm using kafka-python==2.0.2, and have disabled auto_commit but still if I don't commit through code, offsets are automatically getting committed In ...
I read that since version 2.3 of Spring Kafka enable.auto.commit is false by default because Kafka's native implementation of autocommiting mechanism ...
Step 1. I created two topics with different number of partitions, like so: and Step 2. - attempt 1. Create three consumers to listen on two pr ...
I want to consumer to consume messages in batch based on the length/size of the message batch means batch of 1kb and max poll.record with 100. at max ...
I need to process around 50k records(this number can vary from 100 to 50k max) in same topic. Therefore, i used concurrency feature of kafka.Below is ...
I have a working implementation where I am using one producer topic and getting responses for request from two consumer topics. In some scenarios ,ou ...