ActiveMQ Artemis replication mode cause slow performance for publishing messages

Question

Environment:

• OS: Ubuntu 20.04 LTS 64bit
• JDK: OpenJDK 11.0.12
• ActiveMQ: ActiveMQ Artemis 2.27.1 (also tested with 2.25.0)

master broker.xml (only key portion):

      <ha-policy>
         <replication>
            <master>
               <check-for-live-server>true</check-for-live-server>
            </master>
         </replication>
      </ha-policy>

      <cluster-user>alt_cluster</cluster-user>

      <cluster-password>alt_cluster</cluster-password>

      <broadcast-groups>
         <broadcast-group name="bg-group1">
            <group-address>231.7.7.6</group-address>
            <group-port>9876</group-port>
            <broadcast-period>5000</broadcast-period>
            <connector-ref>artemis</connector-ref>
         </broadcast-group>
      </broadcast-groups>

      <discovery-groups>
         <discovery-group name="dg-group1">
            <group-address>231.7.7.6</group-address>
            <group-port>9876</group-port>
            <refresh-timeout>10000</refresh-timeout>
         </discovery-group>
      </discovery-groups>

      <cluster-connections>
         <cluster-connection name="my-cluster">
            <connector-ref>artemis</connector-ref>
            <message-load-balancing>ON_DEMAND</message-load-balancing>
            <max-hops>1</max-hops>
            <discovery-group-ref discovery-group-name="dg-group1"/>
         </cluster-connection>
      </cluster-connections>

slave broker.xml (only key portion)

      <ha-policy>
         <replication>
            <slave>
               <allow-failback>true</allow-failback>
            </slave>
         </replication>
      </ha-policy>

      <cluster-user>alt_cluster</cluster-user>

      <cluster-password>alt_cluster</cluster-password>

      <broadcast-groups>
         <broadcast-group name="bg-group1">
            <group-address>231.7.7.6</group-address>
            <group-port>9876</group-port>
            <broadcast-period>5000</broadcast-period>
            <connector-ref>artemis</connector-ref>
         </broadcast-group>
      </broadcast-groups>

      <discovery-groups>
         <discovery-group name="dg-group1">
            <group-address>231.7.7.6</group-address>
            <group-port>9876</group-port>
            <refresh-timeout>10000</refresh-timeout>
         </discovery-group>
      </discovery-groups>

      <cluster-connections>
         <cluster-connection name="my-cluster">
            <connector-ref>artemis</connector-ref>
            <message-load-balancing>ON_DEMAND</message-load-balancing>
            <max-hops>1</max-hops>
            <discovery-group-ref discovery-group-name="dg-group1"/>
         </cluster-connection>
      </cluster-connections>

I use Apache NIFI as the producer and set 10 for concurrent task number of PublishJMS processor.

Case 1 (yellow frame area)

ActiveMQ running mode: cluster with replication. Master and slave are all running.

The average speed of publishing message is around 60000 messages/minute.

Even I add the concurrent task count to 40, the publishing speed just increases a little bit (65K/min ~70K/min). pls refer to the purple frame.

Case 2 (green frame area)

ActiveMQ running mode: cluster with replication. Master is running.Slave is stopped.

The average speed of publishing message is around 130000 messages/minute.

Case 3 (red frame area)

ActiveMQ running mode: cluster without replication. Two masters are running

The average speed of publishing message is around 130000 messages/minute.

My question is. I hope to run ActiveMQ Artemis in cluster with replication mode. But how to raise the speed of publishing messages?

Answer 1

Replication is, of course, not free. When the message arrives at the primary broker it must then be replicated (ie copied) to the backup. This requires copying the byte array of the original message and a.network round-trip from the primary to the backup as that message data is sent. Once the primary receives the acknowledgement from the backup that it has received the message (ie the message was successfully replicated) and the message is also successfully persisted to the local disk (both of which are done concurrently) then it will send an acknowledgement back to the original message producer. As you can see, this effectively doubles the amount of work required when sending a single message. However, this is unavoidable if you want to physically replicate data in two places reliably .

To potentially speed up the replication process you should:

1. Ensure that the.network connections between the producer and the primary and the primary and the backup are as fast as possible (ie highest throughput and lowest latency).
2. Optimize the size of your messages. The less data in the message the less data must be replicated.

Generally speaking you can increase the performance of sending messages by:

1. Using transactions to batch multiple sends together. This avoids a round-trip for every message.
2. Using the CompletionListener from JMS 2 to send messages asynchronously but reliably. This allows messages to be sent in a non-blocking fashion, but also allows you to detect sending failures so messages can be resent if necessary.
3. Increasing concurrency. Using just 10 message producers is pretty low.
4. Consuming messages as quickly as they are produced. This prevents the broker from entering paging mode which can further decrease performance when sending messages beyond what replication imposes.

It's worth noting that a single broker on good hardware using a mix of persistent and non-persistent messages can sustain throughput measured in the millions of messages per second.