Hazelcast stores large object cause high latency issue

Question

Our restful web service uses hazelcast 3.4.2. We use IQueue to store byte[] object. One thread puts object in the queue, the other thread gets the object from the queue. We use Jmeter to do some load test with 50 threads. When the object is small (under 10k), it works well, the application response time is always under 50ms and the CPU is low. When the object is larger, for example 90k, the response time will go to 500ms. When the object is 250k, the response time will be 2500ms. At the same time, the CPU uses 60%-80%. The memory is about 60%-80%.

Our test server is AWS m5.large:2 core and 8G. Tomcat allocates 6G memory.

We try to fix the issue with these ways:

1. Upgrade the hazelcast to 3.12.
2. Change backup-count to 0.

The issue is not fixed yet.

Here is high latency hazelcast health monitor in prod, the prod server is 8 cores and 16G memory, tomcat allocate 12G memory:

2019-10-25 15:11:13.078][INFO][com.hazelcast.util.HealthMonitor] [ec1-12]:5701 [dev] [3.4.2] processors=8, physical.memory.total=14.7G, physical.memory.free=183.5M, swap.space.total=0, swap.space.free=0, heap.memory.used=1.8G, heap.memory.free=8.8G, heap.memory.total=10.7G, heap.memory.max=10.7G, heap.memory.used/total=17.31%, heap.memory.used/max=17.31%, minor.gc.count=129321, minor.gc.time=2021051ms, major.gc.count=1224, major.gc.time=165825ms, load.process=80.00%, load.system=79.00%, load.systemAverage=469.00%, thread.count=153, thread.peakCount=198, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.oper ation.size=0, executor.q.priorityOperation.size=0, executor.q.response.size=0, operations.remote.size=8, operations.running.size=2, proxy.count=8, clientEndpoint.count=0, connection.active.count=2, client.connection.count=0, connection.count=2

Answer 1

This issue is not related to Hazelcast. What you are seeing is a typical case of high latencies of serializing/deserializing a large object.

In any system where the data needs to travel from one point to another, the data will have to be serialized at origin, travel through the network and may get deserialized (not in your case, depends on configuration) at destination when sent for storage. When you retrieve, the data will be serialized (if deserialized previously) at origin, traverse through the network and deserialized at the destination. In your case, your app is spending most of its time in ser/des as is evident by CPU usage.

The only ways to reduce latency are: 1. use Hazelcast Serialization, read here - https://docs.hazelcast.org/docs/3.12.4/manual/html-single/index.html#serialization 2. reduce the size of object