Why cassandra write fails even when consistency level is met?

Question

I have cassandra cluster setup with following configurations:-

• Nodes:- 3
• DataCenter:- dc1
• Replication Factor:-3
• Consistency:- LOCAL_QUORUM

I have created a sample spring boot api which recursively inserts in table(testing1) in keyspace(test).

Before performing write operation i have introduced delay(2100ms) in response in one of the nodes using tc command, so that when write operation is performed we get writetimeoutexception as write_request_timeout is set to 2000ms in cassandra.yaml. Cassandra should still perform write operation successfully as required nodes to ack to met consistency level(2) are working without any delay but i am getting following exception:-

com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_QUORUM (2 replica were required but only 1 acknowledged the write).

Why is write failing even when required nodes to ack(2) are up and running?

Answer 1

The write has to complete within the timeout to not get a timeout exception. Regardless of if the nodes are up.

Cassandra should still perform write operation successfully as required nodes to ack to met consistency level(2)

No it shouldn't and wont. If the timeout has exceeded nodes may drop the mutation and rely on hints, read repairs and anti-entropy repairs to resolve it. The state of the mutation is unknown and returned as an error so it can be retried or at least not assumed to be applied within a quorum so that it doesn't break consistency contracts.