Spark stream data from IBM MQ

Question

I want to stream data from IBM MQ. I have tried out this code I found on Github.

I am able to stream data from the Queue but each time it streams, it takes all the data from it. I just want to take the current data that is pushed into the queue. I looked up on many sites but didn't find the correct solution.

In Kafka we had something like KafkaStreamUtils for streaming the near-real-time data. Is there anything similar to that in IBM MQ so that it streams only the latest data?

Answer 1

The sample in the link you provided shows that it calls the following method to recieve from the the IBM MQ:

CustomMQReciever(String host , int port, String qm, String channel, String qn)

If you review CustomMQReciever here you can see that it is only Browsing the messages from the queue. This means the message will still be on the queue and the next time you connect you will receive the same messages:

MQQueueBrowser browser = (MQQueueBrowser) qSession.createBrowser(queue);

If you wanted to remove the messages from the queue you would need to call a method that does consume them from the queue instead of browsing them from the queue. Below is an example of changes to CustomMQReciever.java that should accomplish what you want:

---

Under the initConnection() change the above code to the following to cause it to remove the messages from the queue:

MQMessageConsumer consumer = (MQMessageConsumer) qSession.createConsumer(queue);

Get rid of:

enumeration= browser.getEnumeration();

---

Under receive() change the following:

while (!isStopped() && enumeration.hasMoreElements() )
    {

    receivedMessage= (JMSMessage) enumeration.nextElement();
    String userInput = convertStreamToString(receivedMessage);
    //System.out.println("Received data :'" + userInput + "'");
    store(userInput);
    }

To something like this:

while (!isStopped() && (receivedMessage = consumer.receiveNoWait()) != null))
    {
    String userInput = convertStreamToString(receivedMessage);
    //System.out.println("Received data :'" + userInput + "'");
    store(userInput);
    }