压缩期间的Cassandra连接问题

Question

一旦cassandra在其中一个节点上开始压缩，我们就会看到我们的应用程序无法连接到该节点。 我注意到的是，当在其中一个节点上开始压缩时，该节点的CPU负载变为100％，并且随着压缩的进行和cpu负载变低，我们的应用程序能够连接回cassandra。

我们使用m3.xlarge Amazon EC2实例在同一节点上同时运行Java应用程序和cassandra。 我们的应用程序和cassandra都具有以下jvm设置-Xms2G -Xmx2G -Xmn512M。 Cassandra几乎没有300MB的数据，但是仍然可以看到有很多压缩发生。 这是cassandra列族设置之一的样子，我们大约有20个列族具有几乎相同的设置。 每个列族都有一个二级索引。

bloom_filter_fp_chance=0.010000 AND
  caching='ROWS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'min_threshold': '6', 'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

我们正在运行cassandra社区版本2.0.5和数据映射库https://github.com/valchkou/cassandra-driver-mapping 。

我环顾四周，没有看到有人遇到这种问题。 任何帮助将不胜感激。

错误堆栈跟踪

2015-07-29 18:22:50.341 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 234 : [] [] [] Defuncting connection to /172.31.22.125:9042
com.datastax.driver.core.TransportException: [/172.31.22.125:9042] Channel has been closed
    at com.datastax.driver.core.Connection$Dispatcher.channelClosed(Connection.java:636) [cassandra-driver-core-2.1.0.jar!/:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:375) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.0.Final.jar!/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2015-07-29 18:22:50.341 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Cluster 1370 : [] [] [] Host /172.31.22.125:9042 is Suspected
2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-171] closing connection
2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #5] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-172] closing connection
2015-07-29 18:22:50.342 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Connection 103 : [] [] [] [/172.31.22.125:9042-173] Error connecting to /172.31.22.125:9042 (Connection refused: /172.31.22.125:9042)
2015-07-29 18:22:50.342 DEBUG 29056 --- [New I/O worker #6] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-173] closing connection
2015-07-29 18:22:50.343 DEBUG 29056 --- [New I/O boss #9] com.datastax.driver.core.Connection 608 : [] [] [] [/172.31.22.125:9042-173] connection error
java.net.ConnectException: Connection refused: /172.31.22.125:9042
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_75]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.0.Final.jar!/:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Connection 234 : [] [] [] Defuncting connection to /172.31.22.125:9042
com.datastax.driver.core.TransportException: [/172.31.22.125:9042] Cannot connect
    at com.datastax.driver.core.Connection.<init>(Connection.java:104) [cassandra-driver-core-2.1.0.jar!/:na]
    at com.datastax.driver.core.Connection$Factory.open(Connection.java:445) [cassandra-driver-core-2.1.0.jar!/:na]
    at com.datastax.driver.core.Cluster$Manager$4.runMayThrow(Cluster.java:1405) [cassandra-driver-core-2.1.0.jar!/:na]
    at com.datastax.driver.core.ExceptionCatchingRunnable.run(ExceptionCatchingRunnable.java:32) [cassandra-driver-core-2.1.0.jar!/:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.net.ConnectException: Connection refused: /172.31.22.125:9042
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_75]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.0.Final.jar!/:na]
    at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.0.Final.jar!/:na]
    ... 3 common frames omitted
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Cluster 1428 : [] [] [] Host /172.31.22.125:9042 is DOWN
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-36] c.d.driver.core.ControlConnection 592 : [] [] [] [Control connection] /172.31.22.125:9042 is down, currently connected to /172.31.27.168:9042
2015-07-29 18:22:50.343 DEBUG 29056 --- [Cassandra Java Driver worker-37] com.datastax.driver.core.Cluster 1428 : [] [] [] Host /172.31.22.125:9042 is DOWN
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-36] com.datastax.driver.core.Cluster 1473 : [] [] [] /172.31.22.125:9042 is down, scheduling connection retries
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] c.d.driver.core.ControlConnection 592 : [] [] [] [Control connection] /172.31.22.125:9042 is down, currently connected to /172.31.27.168:9042
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-36] c.d.d.core.AbstractReconnectionHandler 60 : [] [] [] First reconnection scheduled in 1000ms
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] com.datastax.driver.core.Cluster 1473 : [] [] [] /172.31.22.125:9042 is down, scheduling connection retries
2015-07-29 18:22:50.344 DEBUG 29056 --- [Cassandra Java Driver worker-37] c.d.d.core.AbstractReconnectionHandler 60 : [] [] [] First reconnection scheduled in 1000ms
2015-07-29 18:22:51.345 DEBUG 29056 --- [Reconnection-0] com.datastax.driver.core.Connection 103 : [] [] [] [/172.31.22.125:9042-174] Error connecting to /172.31.22.125:9042 (Connection refused: /172.31.22.125:9042)
2015-07-29 18:22:51.345 DEBUG 29056 --- [New I/O worker #7] com.datastax.driver.core.Connection 385 : [] [] [] [/172.31.22.125:9042-174] closing connection
2015-07-29 18:22:51.346 DEBUG 29056 --- [New I/O boss #9] com.datastax.driver.core.Connection 608 : [] [] [] [/172.31.22.125:9042-174] connection error
java.net.ConnectException: Connection refused: /172.31.22.125:9042
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_75]

Answer 1

1. 首先，请确保系统防火墙处于打开或关闭状态。
2. Cassandra服务已启动或已关闭。
3. 放入与cassandra和cassandra映射jar相同的jar版本。