Using Spring Integration with RabbitMQ

Question

I'm in the process of developing a messaging interface for one of our applications. The application is a service which is designed to accept a "Job", do some processing, and return the result (in the form of a File actually).

The idea is to use RabbitMQ as the messaging infrastructure and Spring AMQP to handle protocol specific details.

I do not want to have a tight coupling from my code to Spring AMQP, so I would like to use Spring Integration to hide the messaging api. So basically I want this:

Message sent to RabbitMQ ====> Spring AMQP ====> Spring Integration ====> MyService ====> reply all the way back to RabbitMQ

I'm trying to work out the XML configuration required to wire this together, but I'm having problems with the multiple levels of abstraction and different terminology. Finding a working example that demonstrates Spring Integration on top of Spring AMQP/RabbitMQ has proven to be surprisingly difficult, despite the fact that this sort of setup feels very "best practice" to me.

1) So.. Could some brilliant soul out there take a quick look at this and perhaps push me in the right direction? What do I need and what don't I need? :-)

2) Ideally the queue should be multithreaded, meaning that a taskExecutor should hand off multiple messages to my jobService for parallel processing. What configuration would be required?

 <?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:util="http://www.springframework.org/schema/util"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:rabbit="http://www.springframework.org/schema/rabbit"
    xmlns:int="http://www.springframework.org/schema/integration"
    xmlns:int-amqp="http://www.springframework.org/schema/integration/amqp"
    xmlns:int-stream="http://www.springframework.org/schema/integration/stream"
    xsi:schemaLocation="
    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
    http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
    http://www.springframework.org/schema/rabbit http://www.springframework.org/schema/rabbit/spring-rabbit.xsd
    http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
    http://www.springframework.org/schema/integration/amqp http://www.springframework.org/schema/integration/amqp/spring-integration-amqp.xsd
    http://www.springframework.org/schema/integration/stream http://www.springframework.org/schema/integration/stream/spring-integration-stream.xsd
    ">

    <context:component-scan base-package="com.myprogram.etc" />

    <!-- Messaging infrastructure: RabbitMQ -->

    <bean id="connectionFactory" class="org.springframework.amqp.rabbit.connection.CachingConnectionFactory">
        <constructor-arg value="${ei.messaging.amqp.servername}" />
        <property name="username" value="${ei.messaging.amqp.username}" />
        <property name="password" value="${ei.messaging.amqp.password}" />
    </bean>

    <rabbit:connection-factory id="connectionFactory" />

    <rabbit:admin connection-factory="connectionFactory"/>

    <!-- From RabbitMQ -->

    <int-amqp:inbound-gateway request-channel="fromAMQP" reply-channel="toAMQP" queue-names="our-product-name-queue" connection-factory="connectionFactory"/>

    <!-- Spring Integration configuration -->

    <int:channel id="fromAMQP">
        <!-- Is this necessary?? -->
        <int:queue/>
    </int:channel>

    <!-- JobService is a @Service with a @ServiceActivator annotation -->
    <int:service-activator input-channel="fromAMQP" ref="jobService"/>
</beans>

Answer 1

I'm as much a noob to spring-integration and spring-integration-amqp as you, I suspect, but I did get something working based in part on the one sample project.

For rabbitmq infrastructure, I have the following:

<rabbit:connection-factory id="rabbitConnectionFactory"/>

<rabbit:template id="amqpTemplate" connection-factory="rabbitConnectionFactory"/>

<rabbit:admin connection-factory="rabbitConnectionFactory"/>

<!-- some attributes seemed to be ok with queue name, others required id
  -- so I used both with the same value -->
<rabbit:queue id='test.queue' name='test.queue'/>

<rabbit:direct-exchange name:"my.exchange">
    <rabbit:bindings>
        <rabbit:binding queue="test.queue" key="test.binding"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

To send a message to rabbitmq, I have the following:

<!-- This is just an interface definition, no implementation required
  -- spring will generate an implementation which puts a message on the channel -->
<int:gateway id="backgroundService", 
         service-interface="com.company.BackgroundService"
             default-request-channel="toRabbit"

<int:channel id:"toRabbit"/>

<!-- used amqpTemplate to send messages on toRabbit channel to rabbitmq -->
<int-amqp:outbound-channel-adapter channel:"toRabbit" 
                               amqp-template="amqpTemplate" 
                   exchange-name="my.exchange" 
                   routing-key="test.binding"/>

And to receive messages I have the following:

<int:service-activator input-channel="fromRabbit" 
                       ref="testService" 
                       method="serviceMethod"/>


// from rabbitmq to local channel
<int-amqp:inbound-channel-adapter channel="fromRabbit" 
                                  queue-names="test.queue" 
                                  connection-factory="rabbitConnectionFactory"/>

<int:channel id="fromRabbit"/>

Some caveats - the documentation of amqp integration in spring-integration says it is possible to to a synchronous send and receive of a return value, but I haven't figured that out yet. When my service-activator method returned a value, it caused an exception to get thrown, putting the message back on rabbitmq (and generating an infinite loop, since it would then receive the message again and throw the exception again).

My BackgroundService interfacde looks like this:

package com.company

import org.springframework.integration.annotation.Gateway

public interface BackgroundService {

    //@Gateway(requestChannel="someOtherMessageChannel")
    public String sayHello(String toWho)

}

You can specify a channel on every method via the annotation if you don't wish to use the default channel configured in the spring bean.

The service attached to the service-activator looks like this:

package com.company;

class TestService {

    public void serviceMethod(String param) {
    log.info("serviceMethod received: " + param");
    //return "hello, " + param;
    }
}

When I had everything wired up locally without rabbitmq involved, the return value was correctly received by the caller. When I went to rabbitmq channels, I got the aforementioned infinite loop when an exception was thrown after returning a value. It is surely possible or else it wouldn't be possible to wire in different channels without modifying code, but I'm not sure what the trick is yet. Please respond with a solution if you figure it out. Obviously, you can put whatever routing, transforming, and filtering you like between the endpoints, as needed.

Don't be surprised if my XML excerpts above have typos in them. I had to convert back to xml from groovy DSL, so I could have made mistakes. But the intent should be clear enough.