超时直到重启

Question

I set up a cassandra cluster with 2 nodes. After a couple thousand of writes, it starts throwing TimeOut exceptions that don't go away until after I restart the cassandra services. I am using phpcassa (latest at time of writing).

Cassandra.yaml both

• cluster_name: 'Test Cluster'
• seed_provider:
• • class_name: org.apache.cassandra.locator.SimpleSeedProvider
• parameters:
• • seeds: "192.168.2.101"
• rpc_address: 0.0.0.0
• rpc_port: 8080

Cassandra.yaml 1

• initial_token: 0
• listen_address: 192.168.2.101

Cassandra.yaml 2

• initial_token: 85070591730234615865843651857942052864
• listen_address: 192.168.2.102

Assuming this configuration is correct, there must be an issue with my code. Here's the test script I made to reproduce the error:

$pool = Config::ConnectionPool();
$stressTest = new ColumnFamily($pool, 'TwitterTest');
$id = 392766928478932992;

while(true)
{
    try
    {
        $stressTest->insert($id++, array("analyzed"=>0, "followersCount"=>"35",
            "friendsCount"=>"30", "lang"=>"en", "listedCount"=>"0",
            "name"=>"Henky Tanky", "statusesCount"=>"X", "text"=>"@HERPDERP dude i kno i lost a 16gb flash drive #MEH",
            "time"=>"2013-10-22 23:38:27", "twitterId"=>"392766928478932992", "username"=>"mehzor"),
            null, null, \cassandra\ConsistencyLevel::ANY);
    }
    catch (Exception $ex){
        print_r($ex);
        exit;
    }
}

class Config {
    static function ConnectionPool(){
        return new ConnectionPool('KSTwit', array('192.168.2.101:8080', '192.168.2.102:8080'));
    }
}

With that config I get a 'cassandra\\TimedOutException' (Stacktrace 1)

// Set max-retries to 5, and read/write timeout to 60 seconds
new ConnectionPool('KSTwit', array('192.168.2.101:8080', '192.168.2.102:8080'), NULL, 5, 60000, 60000);

With that config I get a 'Thrift\\Exception\\TTransportException' (Stacktrace 2)

I have tried various ConsistencyLevels; ONE, QUORUM and ANY. Same issue with all of them.

As I said, the exceptions don't occur until after a couple throusand of writes. However, if I restart the script after an exception ocurred it will throw exceptions immediately. I have to restart the cassandra services and then it can take a couple thousand again. This only seems to occur with writes. It also occurs if I don't use counter columns so it's not an issue with that.

Stack trace 1

Error performing add on 192.168.2.101:8080: exception 'cassandra\TimedOutException' in /var/installstuff/cassphp/lib/Thrift/Base/TBase.php:206
Stack trace:
#0 /var/installstuff/cassphp/lib/cassandra/Cassandra.php(3575): Thrift\Base\TBase->_read('Cassandra_add_r...', Array, Object(Thrift\Protocol\TBinaryProtocolAccelerated))
#1 /var/installstuff/cassphp/lib/cassandra/Cassandra.php(768): cassandra\Cassandra_add_result->read(Object(Thrift\Protocol\TBinaryProtocolAccelerated))
#2 /var/installstuff/cassphp/lib/cassandra/Cassandra.php(728): cassandra\CassandraClient->recv_add()
#3 [internal function]: cassandra\CassandraClient->add('All Tweets', Object(cassandra\ColumnParent), Object(cassandra\CounterColumn), 1)
#4 /var/installstuff/cassphp/lib/phpcassa/Connection/ConnectionPool.php(264): call_user_func_array(Array, Array)
#5 /var/installstuff/cassphp/lib/phpcassa/ColumnFamily.php(44): phpcassa\Connection\ConnectionPool->call('add', 'All Tweets', Object(cassandra\ColumnParent), Object(cassandra\CounterColumn), 1)
#6 /var/installstuff/cassphp/examples/db/TweetDatabase.php(103): phpcassa\ColumnFamily->add('All Tweets', 'Total')
#7 /var/installstuff/cassphp/examples/core/Helper.php(16): {closure}()
#8 /var/installstuff/cassphp/examples/db/TweetDatabase.php(104): Helper::DoForgiving(Object(Closure))
#9 /var/installstuff/cassphp/examples/core/twitterParser.php(98): TweetDatabaseCassandra->AddTweet('392633738364190...', Array)
#10 /var/installstuff/cassphp/examples/core/twitterParser.php(192): TwitterParser->RunOnce()
#11 /var/installstuff/cassphp/examples/runners/twitterParserCassandra.php(19): TwitterParser->Run()
#12 {main}

Stack trace 2

Error performing add on 192.168.2.101:8080: exception 'Thrift\Exception\TTransportException' with message 'TSocket: timed out reading 4 bytes from 192.168.2.101:8080' in /var/installstuff/cassphp/lib/Thrift/Transport/TSocket.php:284
Stack trace:
#0 /var/installstuff/cassphp/lib/Thrift/Transport/TTransport.php(74): Thrift\Transport\TSocket->read(4)
#1 /var/installstuff/cassphp/lib/Thrift/Transport/TFramedTransport.php(139): Thrift\Transport\TTransport->readAll(4)
#2 /var/installstuff/cassphp/lib/Thrift/Transport/TFramedTransport.php(106): Thrift\Transport\TFramedTransport->readFrame()
#3 /var/installstuff/cassphp/lib/Thrift/Transport/TTransport.php(74): Thrift\Transport\TFramedTransport->read(4)
#4 /var/installstuff/cassphp/lib/Thrift/Protocol/TBinaryProtocol.php(305): Thrift\Transport\TTransport->readAll(4)
#5 /var/installstuff/cassphp/lib/Thrift/Protocol/TBinaryProtocol.php(197): Thrift\Protocol\TBinaryProtocol->readI32(NULL)
#6 /var/installstuff/cassphp/lib/cassandra/Cassandra.php(760): Thrift\Protocol\TBinaryProtocol->readMessageBegin(NULL, 0, 0)
#7 /var/installstuff/cassphp/lib/cassandra/Cassandra.php(728): cassandra\CassandraClient->recv_add()
#8 [internal function]: cassandra\CassandraClient->add('All Tweets', Object(cassandra\ColumnParent), Object(cassandra\CounterColumn), 1)
#9 /var/installstuff/cassphp/lib/phpcassa/Connection/ConnectionPool.php(264): call_user_func_array(Array, Array)
#10 /var/installstuff/cassphp/lib/phpcassa/ColumnFamily.php(44): phpcassa\Connection\ConnectionPool->call('add', 'All Tweets', Object(cassandra\ColumnParent), Object(cassandra\CounterColumn), 1)
#11 /var/installstuff/cassphp/examples/db/TweetDatabase.php(103): phpcassa\ColumnFamily->add('All Tweets', 'Total')
#12 /var/installstuff/cassphp/examples/core/Helper.php(17): {closure}()
#13 /var/installstuff/cassphp/examples/db/TweetDatabase.php(104): Helper::DoForgiving(Object(Closure))
#14 /var/installstuff/cassphp/examples/core/twitterParser.php(98): TweetDatabaseCassandra->AddTweet('392642135327264...', Array)
#15 /var/installstuff/cassphp/examples/core/twitterParser.php(192): TwitterParser->RunOnce()
#16 /var/installstuff/cassphp/examples/runners/twitterParserCassandra.php(19): TwitterParser->Run()
#17 {main}

The cassandra log doesn't really show anything interesting. Except for this ocurring a lot, but that happens even when the exceptions don't occur so I don't think it's the issue;

INFO 10:58:58,241 Timed out replaying hints to /192.168.2.102; aborting further deliveries
WARN 11:07:17,979 MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0.  Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE; upgrade to the Sun JRE instead

Keyspace is setup using 'SIMPLE_STRATEGY' and 'replication_factor=2' According to the 'nodetool ring' command on both machines, the nodes are functioning 'normal' even after the exceptions have been triggered. I honestly don't know what to try next, can anyone spot the issue?

Answer 1

I fixed the issue in one of two ways (can't be sure.)

I initially installed cassandra using a guide online which listed these repo's;

deb http://www.apache.org/dist/cassandra/debian 11x main
deb-src http://www.apache.org/dist/cassandra/debian 11x main

These are outdated, there's a 20x version, so I updated to that.

I also replaced jsvc with sun's java. It's all working fine now.