Cassandra timeout during read query at consistency ONE/LOCAL_QUORUM

Question

Table Structure

CREATE TABLE tablename(
col1 text,
col2 text,
col3 timestamp,
col4 timestamp,
col5 text,
col6 timestamp,
.
.
PRIMARY KEY (col5, col6))
WITH CLUSTERING ORDER BY (col6 DESC)

CREATE CUSTOM INDEX indexname on tablename (col1) USING 'StorageAttachedIndex';
CREATE CUSTOM INDEX indexname on tablename (col2) USING 'StorageAttachedIndex';
CREATE CUSTOM INDEX indexname on tablename (col3) USING 'StorageAttachedIndex';
CREATE CUSTOM INDEX indexname on tablename (col4) USING 'StorageAttachedIndex';
CREATE CUSTOM INDEX indexname on tablename (col6) USING 'StorageAttachedIndex';

Read Query:

select col1, col2, col3, col4, col.... from tablename
where col1='text'
and col2='text'
and col3>'timestamp'
and col4>='timestamp'
and col4<='timestamp'
PER PARTITION LIMIT 1;

In Java, I have written a code to execute a query to fetch 100,000 records with below config:

1. executeAsync
2. Fetch_Size = 10000
3. Not using ALLOW FILTERING
4. DSE - 6.8.9
5. Cql - 3.4.5
6. Cassandra - 4.0.0.681
7. Java driver - 4.6.1

When I run the code, it works perfectly and responding in around 1 min 20 sec for 100,000 rows.

But when I try to run in more than 2 windows parallelly, then only one window showing the result and other windows throwing timeout error.

Cassandra timeout during read query at consistency ONE

Answer 1

When I run the code, it works perfectly and responding in around 1 min 20 sec

TBH I'm surprised this returns a result set at all. Cassandra was not designed to support OLAP or queries requiring filtering on many different columns.

The reason it's timing out, is that queries based on a secondary index (or multiple indexes, in this case) put extra stress on one node. When they run, a "coordinator" node is selected. That node is then responsible for pulling data from all of the other nodes and assembling the result set (in RAM).

The default timeouts are set with the specific intent of stopping queries like this, because they can (and often do) cause nodes to crash. I imagine that supporting two similar queries in parallel is too much for the cluster to handle.

The way around this, is to ensure that your queries are always filtering on a partition key ( col5 in this case). Single partition queries ensure that only a single node will be queried. That's why the idea with Cassandra is to build your tables around the intended queries. In this case, building a query table with partition keys of col1 and col2 would help to ensure that. Adding clustering keys of col3 and col4 will help for your other conditions:

PRIMARY KEY ((col1, col2),col3,col4)

Of course, I'm building that definition without an understanding of the cardinality of col1 or col2 . As Cassandra has a partition limit of 2GB and 2 billion cells, it's always a good idea to keep your partition sizes much lower than that. In which case, an additional partition key and running more than one query for smaller parts of the data set would be the way to go.

I recommend checking out DataStax Academy , specifically the (free) course DS220 on Data Modeling.