SingleKeyBackupWriteCommand 内存不足

Question

Our System has 8 nodes(vm in openstack) to run infinispan. It works well at beginning. However, 7 days later, the SingleKeyBackupWriteCommands run out of one node's memory in 4 hours(Others' memory are fine.). Does Anyone know what's the problem that make infinispan just receives the cmd but never execute them?

 num     #instances         #bytes  class name
----------------------------------------------
   1:      16830203    13201657008  [B
   2:      29186377     3822907360  [C
   3:      20916364     1505978208  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
   4:      30043274      721038576  java.lang.String
   5:     **5750667**    506058696  org.infinispan.commands.triangle.SingleKeyBackupWriteCommand

Answer 1

base on the TriangleSequencer's logic

If any issue happens once in sender side/Network side, the receiver side will never execute any subsequent cmd. It's very fragile logic in TriangleOrderManager.

enter image description here

private synchronized boolean isNext(int commandTopologyId, long sequenceNumber) {
if (receiverTopologyId == commandTopologyId) {
if (trace)

{ log.tracef("Receiver old topology. Current sequence (%s:%s), command sequence (%s:%s)", receiverTopologyId, receiverSequenceNumber, commandTopologyId, sequenceNumber); }
**return receiverSequenceNumber == sequenceNumber;**
} else if (receiverTopologyId < commandTopologyId) {
//update topology. this command will be the first
if (trace)

{ log.tracef("Receiver update topology. CommandTopologyId=%s, command sequence=%s", commandTopologyId, sequenceNumber); }
receiverTopologyId = commandTopologyId;
receiverSequenceNumber = 1;
return 1 == sequenceNumber;
} else {
if (trace)

{ log.tracef("Receiver old topology. Current sequence (%s:%s), command sequence (%s:%s)", receiverTopologyId, receiverSequenceNumber, commandTopologyId, sequenceNumber); }
//this topology is higher than the command topology id.
//another topology was installed. this command will fail with OutdatedTopologyException.
return true;
}
}

More details can be found in https://issues.redhat.com/browse/ISPN-12435