Couchbase Cluster: one node down => entire cluster down?

Question

I'm testing Couchbase Server 2.5`. I have a cluster with 7 nodes and 3 replicates. In normal condition, the system works fine.

But I failed with this test case: Couchbase cluster's serving 40.000 ops and I stop couchbase service on one server => one node down. After that, entire cluster's performance is decreased painfully. It only can server below 1.000 ops. When I click fail-over then entire cluster return healthy.

I think when a node down then only partial request is influenced. Is that right?

And in reality, when one node down, it will make a big impact to entire cluster?

Updated:

I wrote a tool to load test use spymemcached. This tool create multi-thread to connect to Couchbase cluster. Each thread Set a key and Get this key to check immediately, if success it continues Set/Get another key. If fail, it retry Set/Get and by pass this key if fail in 5 times.

This is log of a key when I Set/Get fail.

2014-04-16 16:22:20.405 INFO net.spy.memcached.MemcachedConnection: Reconnection due to exception handling a memcached operation on {QA sa=/10.0.0.23:11234, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 2660829 Key: test_key_2681412 Cas: 0 Exp: 0 Flags: 0 Data Length: 800, topWop=null, toWrite=0, interested=1}. This may be due to an authentication failure. OperationException: SERVER: Internal error at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:192) at net.spy.memcached.protocol.binary.OperationImpl.getStatusForErrorCode(OperationImpl.java:244) at net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:201) at net.spy.memcached.protocol.binary.OperationImpl.readPayloadFromBuffer(OperationImpl.java:196) at net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:139) at net.spy.memcached.MemcachedConnection.readBufferAndLogMetrics(MemcachedConnection.java:825) at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:804) at net.spy.memcached.MemcachedConnection.handleReadsAndWrites(MemcachedConnection.java:684) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:647) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:418) at net.spy.memcached.MemcachedConnection.run(MemcachedConnection .java:1400) 2014-04-16 16:22:20.405 WARN net.spy.memcached.MemcachedConnection: Closing, and reopening {QA sa=/10.0.0.23:11234, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 2660829 Key: test_key_2681412 Cas: 0 Exp: 0 Flags: 0 Data Length: 800, topWop=null, toWrite=0, interested=1}, attempt 0. 2014-04-16 16:22:20.406 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 1 Opaque: 2660829 Key: test_key_2681412 Cas: 0 Exp: 0 Flags: 0 Data Length: 800 2014-04-16 16:22:20.406 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 2660830 Key: test_key_2681412 Cancelled 2014-04-16 16:22:20.407 ERROR net.spy.memcached.protocol.binary.StoreOperationImpl: Error: Internal error 2014-04-16 16:22:20.407 INFO net.spy.memcached.MemcachedConnection: Reconnection due to exception handling a memcached operation on {QA sa=/10.0.0.24:11234, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 2660831 Key: test_key_2681412 Cas: 0 Exp: 0 Flags: 0 Data Length: 800, topWop=null, toWrite=0, interested=1}. This may be due to an authentication failure. OperationException: SERVER: Internal error at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:192) at net.spy.memcached.protocol.binary.OperationImpl.getStatusForErrorCode(OperationImpl.java:244) at net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:201) at net.spy.memcached.protocol.binary.OperationImpl.readPayloadFromBuffer(OperationImpl.java:196) at net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:139) at net.spy.memcached.MemcachedConnection.readBufferAndLogMetrics(MemcachedConnection.java:825) at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:804) at net.spy.memcached.MemcachedConnection.handleReadsAndWrites(MemcachedConnection.java:684) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:647) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:418) at net.spy.memcached.MemcachedConnection.run(MemcachedConnection .java:1400) 2014-04-16 16:22:20.407 WARN net.spy.memcached.MemcachedConnection: Closing, and reopening {QA sa=/10.0.0.24:11234, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 2660831 Key: test_key_2681412 Cas: 0 Exp: 0 Flags: 0 Data Length: 800, topWop=null, toWrite=0, interested=1}, attempt 0. 2014-04-16 16:22:20.408 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 1 Opaque: 2660831 Key: test_key_2681412 Cas: 0 Exp: 0 Flags: 0 Data Length: 800 2014-04-16 16:22:20.408 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 2660832 Key: test_key_2681412 Cancelled

Answer 1

You should find that 6/7 (ie 85%) of your operations should continue to operate at the same performance. However the 15% of operations which are directed at the vbuckets owned by the now downed node will never complete and likely timeout, and so depending on how your application is handling these timeouts you may see a greater performance drop overall.

How are you benchmarking / measuring the performance?

Update: OP's extra details

I wrote a tool to load test use spymemcached. This tool create multi-thread to connect to Couchbase cluster. Each thread Set a key and Get this key to check immediately, if success it continues Set/Get another key. If fail, it retry Set/Get and by pass this key if fail in 5 times.

The Java SDK is designed to make use of async operations for maximum performance, and this is particularly true when the cluster is degraded and some operations will timeout. I'd suggest starting running in a single thread but using Futures to handle the get after the set. For example:

client.set("key", document).addListener(new OperationCompletionListener() {
    @Override
    public void onComplete(OperationFuture<?> future) throws Exception {
        System.out.println("I'm done!");    
    }
});

This is an extract from the Understanding and Using Asynchronous Operations section of the Java Developer guide.

There's essentially no reason why given the right code your performance with 85% of nodes up shouldn't be close to 85% of the maximum for a short downtime.

Note that if a node is down for a long time then the replication queues on the other nodes will start to back up and that can impact performance, hence the recommendation of using auto-failover / rebalance to get back to 100% active buckets and re-create replicas to ensure any further node failures don't cause data loss.