How does Cassandra guarantee eventual consistency in cross region replication?

Question

I cannot find much documentation about it. The only thing I can find is that when the consistency level is not set to EACH_QUORUM, cross region replication is done asynchronously.

But in asynchronous style, is it possible to lose messages? How does Cassandra handle losing messages?

Answer 1

If you don't use EACH_QUORUM and a destination node which would accept a write is down, then coordinator node is saving writes as "hinted handoffs" .

When destination node becomes available again, coordinator replays hinted handoffs on destination.

For any occasion when hinted handoffs are lost, you have to do run a repair on your cluster.

Also you have to be aware of that storing hints is allowed for maximum of 3 hours by defaults.

For further info see documentation at:

http://www.datastax.com/dev/blog/modern-hinted-handoff http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesHintedHandoff.html

Hope this helps.

Answer 2

When you issue a write in Cassandra, the coordinator sends the write to all online replicas, and then blocks. The duration of the block corresponds to consistency level - if you say "ALL", it blocks until all nodes ack the write. If you use "EACH_QUORUM", it blocks until a quorum of nodes in each datacenter ack the write.

For any replica that didn't ack the write, the coordinator will write a hint, and attempt to deliver that hint later (minutes, hours, no guarantee).

Note, though, that the writes were all sent at the same time - what you don't have is a guarantee as to which were delivered. Your guarantee is in the consistency level.

When you read, you'll do something similar - you'll block until you have an appropriate number of replicas answering. If you write with EACH_QUORUM, you can read with LOCAL_QUORUM and guarantee strong consistency. If you write with QUORUM, you could read with QUORUM. If you write with ONE, you could still guarantee strong consistency if you read with ALL.

To guarantee eventual consistency, you don't have to do anything - it'll eventually get there, as long as you wrote with CL >= ONE (CL ANY isn't really a guarantee).