How to configure Spring SimpleMessageListenerContainer receiveTimeout in order to scale up to a reasonable number of consumers

Question

Use case A backend consuming messages at various rate and inserting the messages in a DB. Today in production my SimpleMessageListenerContainer scales to maxConcurrentConsumers even if is not necessary to handle the traffic rate.

Problem I try to find the proper configuration of spring SimpleMessageListenerContainer in order to let spring scale up/down the number of consumers to the adequate number in order to handle the incoming traffic.

With aa fix injection rate, on a single node rabbitmq I have noticed that the scaling process stabilize at numberOfConsumers = (injectionRate * receiveTimeoutInMilliseconds) / 1000

For example: injection rate: 100 msg/s container.setReceiveTimeout(100L); // 100 ms container.setReceiveTimeout(100L); // 100 ms --> consumers 11 --> Consumer capacity 100%

injection rate: 100 msg/s container.setReceiveTimeout(1000L); // 1 s - default container.setReceiveTimeout(1000L); // 1 s - default --> consumers 101 --> Consumer capacity 100%

Knowing that more consumers means more threads and more amqp channels I am wondering why the scaling algorithm is not linked to the consumerCapacity metric and why is the default receive timeout set to 1 second?

Answer 1

See the documentation https://docs.spring.io/spring-amqp/docs/current/reference/html/#listener-concurrency

In addition, a new property called maxConcurrentConsumers has been added and the container dynamically adjusts the concurrency based on workload. This works in conjunction with four additional properties: consecutiveActiveTrigger , startConsumerMinInterval , consecutiveIdleTrigger , and stopConsumerMinInterval . With the default settings, the algorithm to increase consumers works as follows:

If the maxConcurrentConsumers has not been reached and an existing consumer is active for ten consecutive cycles AND at least 10 seconds has elapsed since the last consumer was started, a new consumer is started. A consumer is considered active if it received at least one message in batchSize * receiveTimeout milliseconds.

With the default settings, the algorithm to decrease consumers works as follows:

If there are more than concurrentConsumers running and a consumer detects ten consecutive timeouts (idle) AND the last consumer was stopped at least 60 seconds ago, a consumer is stopped. The timeout depends on the receiveTimeout and the batchSize properties. A consumer is considered idle if it receives no messages in batchSize * receiveTimeout milliseconds. So, with the default timeout (one second) and a batchSize of four, stopping a consumer is considered after 40 seconds of idle time (four timeouts correspond to one idle detection).

Practically, consumers can be stopped only if the whole container is idle for some time. This is because the broker shares its work across all the active consumers.

So, when you reduce the receiveTimeout you would need a corresponding increase in the idle/active triggers.

The default is 1 second to provide a reasonable compromise between spinning an idle consumer while retaining responsive behavior to a container stop() operation (idle consumers are blocked for the timeout). Increasing it will cause a less responsive container (for stop() ).

It is generally unnecessary to set it lower than 1 second.