How can i properly distribute data into two consumers?

Question

In my application i'm using active mq,at consumer side i'm running two instances(consumers) to process the request.As two instances are listen to the same queue,some conflicts occurred. If same data received multiple times one request processed multiple times,to overcome this and to communicate two instances i have implemented hazelcast and it is working good,but some times data is not distributing properly into two instances,if i send bulk data only one instance is processing all tasks.

The code that i'm using at producer side.

public synchronized static void createMQRequestForPoster(Long zoneId, List<String> postJobs, int priority) throws JMSException {

    Connection connection = null;
    try {
        connection = factory.createConnection();

        connection.start();
        Session session = connection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
        Queue queue = session.createQueue("customQueue");
        MessageProducer producer = session.createProducer(queue);
        producer.setDeliveryMode(DeliveryMode.PERSISTENT);

        logger.info("List of jobs adding into poster queue: "+postJobs.size());
        for(String str : postJobs) {

            TextMessage message = session.createTextMessage();
            JSONObject obj = new JSONObject();
            obj.put("priority", priority);
            obj.put("zoneId", zoneId);
            obj.put("postJobs", str);
            logger.debug(obj.toString());
            message.setText(obj.toString());
            message.setIntProperty(ActiveMQReqProcessor.REQ_TYPE, 0);
            producer.send(message);
        }
    } catch (JMSException | JSONException e) {
        logger.warn("Failed to add poster request to ActiveMq", e);
    } finally {
        if(connection != null)
            connection.close();
    }
}

The code that i'm using at consumer side.

private static void activeMQPostProcessor() {

    ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(AppCoding.NZ_JMS_URL);
    Connection connection = null;
    try {
        connection = connectionFactory.createConnection();
        connection.start();
        Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
        Queue queue = session.createQueue("customQueue");


        MessageConsumer consumer = session.createConsumer(queue);
        MessageListener listener = new MessageListener() {

            @Override
            public void onMessage(Message message) {

                if (message instanceof TextMessage) {

                    try {
                        TextMessage textMessage = (TextMessage) message;
                        logger.info("Received message " + textMessage.getText());
                        JSONObject jsonObj = new JSONObject(textMessage.getText());
                        HazelcastClusterInstance.getInstance().add(processOnPoster(jsonObj));
                        message.acknowledge();
                    } catch (JSONException e) {

                    } catch (JMSException e) {

                    }

                    logger.info("Adding Raw Message to Internal Queue");
                }
            }
        };
        consumer.setMessageListener(listener);
        logger.info("Waiting for new posts from selector, scheduler.");

    } catch (JMSException e) {
        logger.info(e);
    }
}

I am observing this case in only times.How can i handle this?

Answer 1

Most likely you are seeing uneven balance of prefetched messages. The consumer connects and requests a prefetch block and the first one tends to get a large batch before the second one gets in so it seems to hog messages.

You need to tune prefetch for your use case, refer to the prefetch documentation .