Cassandra consistency issue when node dies/offline in between transaction

Question

I have a scenario

Lets assume we have 6 nodes in a cluster with replication factor of 3. Now we are writing with quorum so lets say the coordinator see that 3 nodes are up where data needs to go but as and when it sends the data to 3 nodes n1, n2, n3. n1 and n2 stop working and thus the write operation will fail as quorum is not met but n3 will have the data of this failed upsert because cassandra has no rollback.

After this n1 and n2 come up but having older data.

Now if read is done, in read repair the latest data (of failed upsert) present on n3 will get replicated to n1 and n2 is my understanding correct?

Answer 1

Yes, you are correct.

In the case a read repair runs for that record, the data will be replicated from n3. This will depend on your configuration of read_repair_chance and how often the record is queried.

If the record isn't being queried a lot it's unlikely for a read repair to run and you will have to wait for your repair to run.

If you don't run nodetool repair on a regular schedule, you should start doing so!

Note that if you read with QOURUM consistency before a repair has occurred you will still get the old value.

Answer 2

Theres two different kinds of errors. If n1 and n2 are DOWN the write wont even go to n3 either and you will get an Unavailable exception and no issue.

If n1 and n2 go down AFTER the write started or had some catastrophic loss then the data will still exist on n3 and you will get a WriteTimeoutException as the n3 coordinator will be sitting and waiting for other 2 replicas to respond. On write timeouts you need to handle the error differently, usually its checking with a serial read (if using LWT) or another kinda of read. Usually though the writes are idempotent and you can just try again.

With QUORUM and rf=3 you can only really safely handle one node going down. Once you get 2 replicas down your going to have lots of potential issues up to even data loss. This is why many use multiple DCs and higher replication factors if data is really important, that way losing a DC even wont impact things.