kafka - Asynchronous and acks

Question

In Kafka context, Asynchronous and asks concept are confusing for me, but I would like understand these concepts clearly.

1. If asks = 1 or all, does a Kafka producer need to wait for the ack response from broker and can't do nothing?

2. Without the ack response, Kafka producer can't send next message to broker? If so, it looks like synchronous because the producer is waiting for the ack result from broker.

Could you help me to understand well these concepts?

Thanks.

Answer 1

The answer lies in the send() method.

As per the official docs:

public Future send(ProducerRecord record, Callback callback)

Asynchronously send a record to a topic and invoke the provided callback when the send has been acknowledged. The send is asynchronous and this method will return immediately once the record has been stored in the buffer of records waiting to be sent. This allows sending many records in parallel without blocking to wait for the response after each one.

As you can see the signature - it accepts a callback method as well as returns a Future object.

Upon calling this method, the method itself doesn't really care about the acks config. It won't block the call but will leave the decision to the calling method by returning a Future object and accepting a callback as well. It'll push messages to buffer as it receives, and leaves the rest to these two ie Future and callback.

At the buffer level, the acks starts getting honored, but this is done parallel and doesn't block the send caller.

With acks=0, the producer will assume the message is written as it gets sent. (aka Fire and forget)

With acks=1, the producer will consider the write successful only when the leader receives the record, in case of no acknowledgment - it'll retry based on your configuration and use the callback accordingly.

With acks=all -> This only changes the acknowledgment part - ie the producer will consider the write successful successfully written on all replicas, as it'll receive acknowledgment based on min.insync.replicas . Rest is as acks=1.

With the future you received, you can either check it later and continue sending messages or call the get() method - which would cause a block.

Or you can use a callback to perform the action as and when acknowledgement is received.

So TLDR;

If asks = 1 or all, does a Kafka producer need to wait for the ack response from broker and can't do nothing?

Depends on whether you instantly use the Future.get() method - which would block.

producer.send(msg).get() //Waiting...

Or just ignore the return and delegate actions to a callback

producer.send(record,
           new Callback() {
                   // Do the same thing
               }
           });

Answer 2

acks = 1 or all does not mean that the producer will not send the next batch until it gets the acknowledgement of the previous batch; it means that if it does not get an acknowledgement from the leader (acks = 1) or leader and all the ISRs (acks = all) - it will attempt a retry.

Even without acknowledgement of the previous batch - producers will send the next batch given you have set the configurations for it (like max.in.flight.requests.per.connection is greater than one and you implement callback on the producer side for processing the response in the asynchronous way.)