Cassandra read performance with Astyanax client

Question

We are using Cassandra database in production environment . We have a single cross colo cluster of 24 nodes meaning 12 nodes in PHX and 12 nodes in SLC colo . We have a replication factor of 4 which means 2 copies will be there in each datacenter .

Below is the way by which keyspace and column families have been created by our Production DBA's .

create keyspace profile with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {slc:2,phx:2};

 create column family PROFILE_USER with key_validation_class = 'UTF8Type' and comparator = 'UTF8Type' and default_validation_class = 'UTF8Type' and gc_grace = 86400; 

We are running Cassandra 1.2.2 and it has org.apache.cassandra.dht.Murmur3Partitioner , with KeyCaching , SizeTieredCompactionStrategy and Virtual Nodes enabled as well. Cassandra nodes is deployed on HDD instead of SSD's`.

I am using Astyanax client to read the data from Cassandra database using consistency level as ONE . I inserted 50 Millions records (total around 285GB of data across 24 nodes) in the production cluster using the Astyanax client and after the compaction is finished, I started doing read against the Cassandra production database .

Below is the code by which I am creating connection configuration using Astyanax client -

/**
 * Creating Cassandra connection using Astyanax client
 *
 */
private CassandraAstyanaxConnection() {

    context = new AstyanaxContext.Builder()
    .forCluster(ModelConstants.CLUSTER)
    .forKeyspace(ModelConstants.KEYSPACE)
    .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
        .setPort(9160)
        .setMaxConnsPerHost(100)
        .setSeeds("cdb03.vip.phx.host.com:9160,cdb04.vip.phx.host.com:9160")
        .setLocalDatacenter("phx") //filtering out the nodes basis on data center
    )
    .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
        .setCqlVersion("3.0.0")
        .setTargetCassandraVersion("1.2")
        .setConnectionPoolType(ConnectionPoolType.ROUND_ROBIN)
        .setDiscoveryType(NodeDiscoveryType.RING_DESCRIBE))
    .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
    .buildKeyspace(ThriftFamilyFactory.getInstance());

    context.start();
    keyspace = context.getEntity();

    emp_cf = ColumnFamily.newColumnFamily(
        ModelConstants.COLUMN_FAMILY, 
        StringSerializer.get(), 
        StringSerializer.get());
}

Most of the time I am getting 95th percentile read performance around 8/9/10 ms .

I am trying to see is there any way I can get much better read performance with Cassandra database . I was in the impression that I will be getting 95th percentile as 1 or 2 ms but after doing some tests on the production cluster all my hypothesis went wrong. Ping time to Cassandra production nodes from where I am running my client program is 0.3ms average .

Below is the result I am getting.

Read Latency(95th Percentile)      Number of Threads    Duration the program was running(in minutes)    Throughput(requests/seconds)    Total number of id's requested    Total number of columns requested
    8 milliseconds                         10                      30                                               1584                              2851481                        52764072

Can anyone shed some light on what other things I can try it out to achieve good read latency performance? I know there might be similar people in my same situation as well who are using Cassandra in production. Any help will be appreciated.

Thanks for the help.

Answer 1

I'd try the following:

Astyanax

Set the ConnectionPoolType to TOKEN_AWARE instead of ROUND_ROBIN.

Additionally, I'd use some of the Astyanax latency aware connection pool features. For example:

.withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
        .setPort(9160)
        .setMaxConnsPerHost(100)
        .setSeeds("cdb03.vip.phx.host.com:9160,cdb04.vip.phx.host.com:9160")
        .setLocalDatacenter("phx") //filtering out the nodes basis on data center
        .setLatencyScoreStrategy(new SmaLatencyScoreStrategyImpl(10000,10000,100,0.50))
    )

The latency setting are provided via the constructor of the ScoreStrategy. eg SmaLatencyScoreStrategyImpl .

I am in the process of figuring this out as well, so I'll post back here if I learn anything additional.

See: Latency and Token Aware configuration

Cassandra

You could do a couple of things to optimize reads. Note: I have not tried these, but they are on my list of things to investigate (so I figured I'd share).

Cache

Enable the Key cache and Row cache.

KeyCache

bin/nodetool --host 127.0.0.1 --port 8080 setcachecapacity MyKeyspace MyColumnFam 200001 0

RowCache

bin/nodetool --host 127.0.0.1 --port 8080 setcachecapacity MyKeyspace MyColumnFam 0 200005

Then check the hit rates after banging on that node for a while with your app scenarios:

bin/nodetool --host 127.0.0.1  --port 8080 cfstats

Consistency

Consider Read Consistency to ONE See this on Data Consistency (this is DataStax docs but still relevant)

Consider lowering the read repair chance.

update column family MyColumnFam with read_repair_chance=.5

After lowering the read_repair_chance consider tweaking the replication factor to help with read performance (but this will kill writes as we'll be writing to more nodes).

create keyspace cache with replication_factor=XX;

Disk

Not sure if there is anything to be done here but thought I should include it. Ensure optimal file system (eg ext4). If you have a high replication factor we could optimize the disk around that (knowing that we'll get our durability from Cassandra). ie what RAID level is best for our setup.