Cassandra Read/Get Performance

Question

My Cassandra table has following schema

CREATE TABLE cachetable1 (
id text,
lsn text,
lst timestamp,
PRIMARY KEY ((id))
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='{"keys":"ALL", "rows_per_partition":"ALL"}' AND
comment='' AND
dclocal_read_repair_chance=0.100000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.000000 AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};

Above table contains 221 Million rows (approx. 16 GB data). The CassandraDaemon is running with 4GB heap space and I have configured 4 GB memory for row cache. I am try to run select queries from my java code like this

for(int i = 0; i < 1000; i ++)
    {
        int id = random.nextInt(20000000 - 0) + 0;
        for(j = id; j <= id + 100; j++)
        {
            ls.add(j+"");
        }

           Statement s = QueryBuilder.select("lst","lsn").from("ks1" , "cachetable1").where(QueryBuilder.in("id",ls.toArray()));
           s.setFetchSize(100);

               ResultSet rs=sess.execute( s );
              List<Row> lsr=rs.all();
               for(Row rw:lsr)
               {
                   //System.out.println(rw.toString());
                   count++;
               }

        ls.clear();
    }

In above code, I am trying to fetch 0.1 Million records. But the read/get performance is very bad. It takes 400-500 seconds to fetch 0.1 Million rows. Is there any better way to read/get records from Cassandra through Java? Is some tuning required other than row cache size and Cassandra heap size?

Answer 1

You appear to want to retrieve your data in 100 row chunks. This sounds like a good candidate for a clustering column.

Change your schema to use an id as the partition key and a chunk index as a clustering column, ie PRIMARY KEY ( (id), chunk_idx ). When you insert the data, you will have to figure out how to map your single indexes into an id and chunk_idx (eg perhaps do a modulo 100 on one of your values to generate a chunk_idx).

Now when you query for an id and don't specify a chunk_idx, Cassandra can efficiently return all 100 rows with one disk read on the partition. And you can still do range queries and retrievals of single rows within the partition by specifying the chunk_idx if you don't always want to read a whole chunk of rows.

So your mistake is you are generating 100 random partition reads with each query, and this will hit all the nodes and require a separate disk read for each one. Remember that just because you are querying for sequential index numbers doesn't mean the data is stored close together, and with Cassandra it is exactly the opposite, where sequential partition keys are likely stored on different nodes.

The second mistake you are making is you are executing the query synchronously (ie you are issuing the query and waiting for the request to finish before you issue any more queries). What you want to do is use a thread pool so that you can have many queries running in parallel, or else use the executeAsync method in a single thread. Since your query is not efficient, waiting for the 100 random partition reads to complete is going to be a long wait, and a lot of the highly pipelined Cassandra capacity is going to be sitting there twiddling its thumbs waiting for something to do. If you are trying to maximize performance, you want to keep all the nodes as busy as possible.

Another thing to look into is using the TokenAwarePolicy when connecting to your cluster. This allows each query to go directly to a node that has a replica of the partition rather than to a random node that might have to act as a coordinator and get the data via an extra hop. And of course using consistency level ONE on reads is faster than higher consistency levels.

The row cache size and heap size are not the source of your problem, so that's the wrong path to go down.

Answer 2

I am going to guess that this is your culprit:

.where(QueryBuilder.in("id",ls.toArray()))

Use of the IN relation in the WHERE clause is widely known to be non-performant. In some case, performing many parallel queries can be faster than using one IN query. From the DataStax SELECT documentation :

When not to use IN

...Using IN can degrade performance because usually many nodes must be queried. For example, in a single, local data center cluster with 30 nodes, a replication factor of 3, and a consistency level of LOCAL_QUORUM, a single key query goes out to two nodes, but if the query uses the IN condition, the number of nodes being queried are most likely even higher, up to 20 nodes depending on where the keys fall in the token range.

So you have two options (assuming that living with this poor-performing query isn't one of them):

1. Rewrite your code to make multiple, parallel requests for each id .

2. Revisit your data model to see if you have another value that it makes sense to key your data by. For instance, if all of your id s in ls happen to share a common column value that is unique to them, that's a good candidate for a primary key. Basically, find another way to query all of the id s that you are looking for, and build a specific query table to support that.