multiple consumers with latest and earliest offset options with spring-kafka

Question

There's one consumer with earliest offset option and batch processing which works perfectly fine.

Now there's need to add another consumer which should work in single processing mode and always look for latest offset - this one appears to be in clash with some of configs below because it doesn't really have messages processed.

properties for whole application:

spring.kafka.bootstrap-servers=${KAFKA_ADDRESS:localhost:9092}
spring.kafka.consumer.group-id=group
spring.kafka.listener.type=batch
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.max-poll-records=100000

bean for working consumer:

@Bean("batchKafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<?, ?> batchKafkaListenerContainerFactory(ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> kafkaConsumerFactory) {
    ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
    configurer.configure(factory, kafkaConsumerFactory);
    factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
    factory.setBatchListener(true);
    return factory;
}

bean for problem consumer:

@Bean("singleKafkaListenerContainerFactoryManualCommit")
public ConcurrentKafkaListenerContainerFactory<?, ?> singleKafkaListenerContainerFactoryManualCommit(ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> kafkaConsumerFactory) {
    ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
    configurer.configure(factory, kafkaConsumerFactory);
    Properties props = new Properties();
    props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
    factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
    factory.getContainerProperties().setKafkaConsumerProperties(props);
    factory.setBatchListener(false);
    return factory;
}

Both consumers are up and running as follows:

[ntainer#1-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-group-2, groupId=group] Successfully joined group with generation 15
[ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-group-1, groupId=group] Successfully joined group with generation 15

they do target same topic but assuming that there are separate consumer groups in play this shouldn't affect anything in this case.

Maybe there's tricky problem of consumer waiting to poll entire batch or another option interference plays out?

UPD Based on comments this can be done via plain and simple separate consumer group addition.

This is how I've figured to amend properties for new consumer group creation:

props.put("group.id", "group-single");

and another option:

props.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "group-single");

Still logs indicate no presence of new consumer however.

Answer 1

This one turns out to be solved by adding this one instead of props map:

factory.getContainerProperties().setClientId("consumer-group-other-1");
factory.getContainerProperties().setGroupId("consumer-group-other");