简体   繁体   中英

Couchbase BucketClosedException while bucket is being used actively

CB server- Enterprise Edition 6.0.0 build 1693
CB Java sdk- 2.7.10

I'm running stress test on my CB cluster, where I'm generating load of up to 1k ops/sec. The load is a mix of set/get operations. The 2 operations are fairly equal in number. The cluster has 2 buckets, of which I'm using only one.
I'm connecting to the cluster in following manner

Cluster cluster = CouchbaseCluster.create(config.getString(CB_GE_HOSTS));
        cluster.authenticate(new PasswordAuthenticator(config.getString(CB_GE_USERNAME), config.getString(CB_GE_PASSWORD)));
        bucket = cluster.openBucket(config.getString(CB_GE_BUCKET));


In my code, i do bucket.disconnect() only when the pod get a shutdown signal. so I'm not explicitly closing the bucket or the cluster connection.
While the operations succeed initially, I start to get BucketClosedException while the operations are running.
Following is the stacktrace I've in the logs-

com.couchbase.client.core.BucketClosedException: gameengine has been closed
        at com.couchbase.client.core.RequestHandler.dispatchRequest(RequestHandler.java:240) ~[com.couchbase.client.core-io-1.7.10.jar:na]
        at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:207) ~[com.couchbase.client.core-io-1.7.10.jar:na]
        at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:78) ~[com.couchbase.client.core-io-1.7.10.jar:na]
        at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) ~[com.couchbase.client.core-io-1.7.10.jar:na]
        at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) ~[com.couchbase.client.core-io-1.7.10.jar:na]
        at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[com.couchbase.client.core-io-1.7.10.jar:na]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_242]


Doesn't this happen usually with idle connections only? I've a single pod instance running, and I'm creating connection to the cluster just once(in static code segment).
This mostly happens more often when I increase the traffic to CB cluster. For some initial requests, CB performs fine, but then all calls to CB start to fail. The cluster currently has a single EC2 instance, with 90GB of data service and 512 MB index service. When I look at server resources, it lists ~45 connections, with a single pod connected to it.

The bucket.close() call was being made explicitly from the client end. This was determined from the memcache logs on CB server.
In one of the flows in the application, a NPE was occuring which was not being logged. When I tailed pod logs (kubectl logs --follow) , I saw that an NPE would occur, sending a restart signal to the pod, and this would initiate shutdown handler in the application, causing the connection to close.
This was consistent with the finding that all calls to CB in a pod would fail after first error while connecting to CB. And it would succeed in next run for same pod since there was a time gap between test runs, giving time to the pod to restart.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM