Distributed Caching use case with Hazelcast

Question

Here is our use-case:

We have 4-8 nodes which implement a distributed hazelcast cache Each nodes can get multiple messages via JMS which have the same transaction key, we add all the messages in the cache with a key-object pair where the object is a List of cacheable items

The aggregation is done via apache-camel where we have a aggregator that runs for 300 MILLISECS After 300 MILLISECS we close the aggregation window and evict all items from the cache corresponding to the transaction key and programmatically aggregate them. Hazelcast's role is only to collect messages in a cache map and keep all nodes in sync where-as all nodes know what items are in the cache Problem scenario:

1. Node 1 adds a item1 in the cache for transaction-key=tx1 at 17:01:01:100 - and we assume that hazelcast broadcasts this information to other nodes such that all nodes get a copy of the cached item.
2. Node 2 adds a item2 in the cache for transaction-key=tx1 at 17:01:01:105
3. Node 3 adds a item3 in the cache for transaction-key=tx1 at 17:01:01:350 Now, assume that at 17:01:01:400 we try to close the aggregation window and programmatically evict all items from the cache using transaction-key=tx1

The problem is that node1, node2 know about item1 and item2 since they have had time to sync their cache items. BUT node3 added item3 so late that node1 and node2 dont know about item3 at 17:01:01:400. Assume we kick-off the eviction at 17:01:01:400 from node1 which does not know about item3 and it will give us a object with only 2 items Then node1 will broadcast to everyone to evict all cache items for transaction-key=tx1 in response to which all nodes will clear their cache for transaction-key=tx1 This leads to a lost item(item3)

Appreciate any help.

Answer 1

I presume your IMap is designed as follows :

IMap<TransactionKey,List<CacheItem>> transactionMessagesMap .

The problem is that node1, node2 know about item1 and item2 since they have had time to sync their cache items. BUT node3 added item3 so late that node1 and node2 dont know about item3 at 17:01:01:400.

How are you putting an entry to the IMap ? Are you using set() , put() or putAsync() ? put() and set() are synchronous calls and hence changes made by node 3 will get reflected in a synchronous fashion.

Note that put() / set() operations on the IMap are synchronous. However if you want to make sure that when one node is operating on your list (for eg adding a new item), all other subsequent requests from the other nodes should see the updated list, then you might need to make use of a distributed lock against the relevant key. ie

The code to add new item may be as follows:

transactionMessagesMap.lock(transactionKey);
list = transactionMessagesMap.get(transactionKey);
list.add(newItem);
transactionMessagesMap.set(transactionKey, list);
transactionMessagesMap.unlock(transactionKey);

The code to evict items may be as follows :

// In your case node1 will block here until node3 has successfully updated 
// item3 in the list.
transactionMessagesMap.lock(transactionKey);
list = transactionMessagesMap.get(transactionKey);
for(CacheItem item : list)
{
   // perform your eviction logic here.
}
transactionMessagesMap.unlock(transactionKey);

You can decide when to release the lock according to your logic. The basic idea is to make use of some distributed locking mechanism like IMap.lock() or ILock according to your use case. Hope this helps.