Handle multiple amqp messages concurrently through one consumer inside one spring-rabbit service

Question

EDIT

Just found out how to run multiple consumers inside one service:

  @Bean
  SimpleMessageListenerContainer container(ConnectionFactory connectionFactory, MessageListenerAdapter listenerAdapter) {
    SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
    container.setConnectionFactory(connectionFactory);
    container.setQueueNames(RENDER_QUEUE);
    container.setConcurrentConsumers(concurrentConsumers); // setting this in env
    container.setMessageListener(listenerAdapter);
    return container;
  }

  @Bean
  MessageListenerAdapter listenerAdapter(RenderMessageConsumer receiver) {
    return new MessageListenerAdapter(receiver, "reciveMessageFromRenderQueue");
  }

Now the only question that remains is: how can I have a global limit? So how do multiple instances of the AMQP receiver share the total number of consumers? So I want to set a global number of concurrentConsumers to 10, run 2 instances of the consumerSerivce and have each instance run around 5 consumers. Can this be managed by rabbitMq?

---

I have a Spring service that consumes AMQP messages and calls a http resource for each message. After the http call completes another queue is called to either report error or done. Only then will message handling complete and the next message be taken from the queue.

  // simplified
  @RabbitListener(queues = RENDER_QUEUE)
  public void reciveMessageFromRenderQueue(String message) {
    try {
      RenderMessage renderMessage = JsonUtils.stringToObject(message, RenderMessage.class);
      String result = renderService.httpCallRenderer(renderMessage);
      messageProducer.sendDoneMessage(result);
    } catch (Exception e) {
      logError(type, e);
      messageProducer.sendErrorMessage(e.getMessage());
    }
  }

There are at times hundreds or thousands of render messages in the queue but the http call is rather long running and not doing much. This becomes obvious as I can improve the message handling rate by running multiple instances of the service thus adding more consumers and calling the http endpoint multiple times. One instance has exactly one consumer for the channel so the number of instances is equal to the number of consumers. However that heavily increases memory usage (since the service uses spring) for just forwarding a message and handling the result.

So I thought, I'd do the http call asynchronously and return immediatly after accepting the message:

.httpCallRendererAsync(renderMessage)
    .subscribeOn(Schedulers.newThread())
    .subscribe(new Observer<String >() {
      public void onNext(String result) {
        messageProducer.sendDoneMessage(result);
      }
      public void onError(Throwable throwable) {
        messageProducer.sendErrorMessage(throwable.getMessage());
      }
    });

That however overloads the http endpoint which cannot deal with 1000 or more simultanous requests.

What I need is for my amqp service to take a certain amount of messages from the queue, handle them in separate threads, make the http call in each of them and return with "message handled". The amount of messages taken from the queue however needs to be shared between multiple instances of that service, so if maximum is 10, message consumption is round robin, the first 5 odd messages should be handled by instance one and the first 5 even messages by instance 2 and as soon as one instance finishes handling the message it should take another one from the queue.

What I found are things like prefetch with limts by consumer and by channel as described by rabbitmq . And the spring-rabbit implementation which uses prefetchCount and the transactionSize described here . That however does not seem to do anything for a single running instance. It will not spawn additional threads to handle more messages concurrently. And of course it will not reduce the number of messages handled in my async scenario since those messages are immediatly considered "handled".

  @Bean
  public RabbitListenerContainerFactory<SimpleMessageListenerContainer> prefetchContainerFactory(ConnectionFactory rabbitConnectionFactory) {
    SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
    factory.setConnectionFactory(rabbitConnectionFactory);
    factory.setPrefetchCount(5);
    factory.setTxSize(5);
    return factory;
  }

  // and then using
  @RabbitListener(queues = RENDER_QUEUE, containerFactory = "prefetchContainerFactory")

The most important requirement for me seems to be that multiple messages should be handled in one instance while the maximum of concurrently handled messages should be shared between instances. Can that be done using rabbitMq and spring? Or do I have to implemenent something in between.

In an early stage it might be acceptable to just have concurrent message handling in one instance and not share that limit. Then I'll have to configure the limit manually using environment variables while scaling the number of instances.

Answer 1

Now the only question that remains is: how can I have a global limit? So how do multiple instances of the AMQP receiver share the total number of consumers? So I want to set a global number of concurrentConsumers to 10, run 2 instances of the consumerSerivce and have each instance run around 5 consumers. Can this be managed by rabbitMq?

There is no mechanism in either RabbitMQ or Spring to support such a scenario automatically. You can, however, change the concurrency at runtime ( setConcurrentConsumers() on the container) so you could use some external agent to manage the concurrency on each instance.