How can I configure (long) timeout for Kafka Admin operations?

Question

I fancy myself making a single request creating 15k topics in a busy Kafka cluster, in a single request, something like this:

final Admin admin = ...;
final List<NewTopic> newTopics = IntStream.range(0, 15000)
    .mapToObj(x -> "adam-" + x)
    .map(x -> new NewTopic(x, Optional.empty(), Optional.empty()))
    .collect(toList());
final CreateTopicsResult ctr = admin.createTopics(newTopics);

ctr.all().get(); // Throws exceptions.

Unfortunately this starts throwing exceptions due to embedded timeouts - how can I properly make the request while keeping it simple without batching?

For the sake of argument let's stick to Kafka 3.2 (both client & server).

Answer 1

This can be configured in one of two ways:

A) Operation timeout allowed the underlying NetworkClient to wait indefinitely until the response is received. So we can change our code to admin.createTopics(newTopics, new CreateTopicOptions().timeout(Integer.MAX_VALUE)) .

B) Alternatively, we could configure Admin's default.api.timeout.ms property and need not to provide the explicit timeout - which one is preferred depends on the codebase/team standards.

What was not necessary:

• request.timeout.ms - it appears to apply to only a single request, however not setting it to large value (larger than expected duration) results in somewhat strange behaviour when original request appears to be failed on finish, and then repeated (to be processed immediately in case of topic-creation, as they'd have been created by initial request (this can be quite easily replicated by setting request.timeout.ms to a low value) ):

Cancelled in-flight CREATE_TOPICS request with correlation id 3 due to node 2 being disconnected (elapsed time since creation: 120443ms, elapsed time since send: 120443ms, request timeout: 29992ms)

• connections.max.idle.ms - the connection remains active the whole time the NC is waiting for the upstream's response.

• metadata.max.age.ms - metadata fetching is needed only for the initial step (figuring out where to send the request), but later we just wait for the response from the known upstream broker.