为什么双节点集群的一个节点宕机时，CQL Batch 语句会失败？

Question

I have created a Cassandra cluster with 2 nodes and keyspaces with a replication factor of 2:

CREATE KEYSPACE data WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };

Everything works fine when both nodes are up. But whenever I take down one of the nodes, I receive the following error from my Java client:

com.datastax.driver.core.exceptions.UnavailableException: Not enough replica available for query at consistency ONE (1 required but only 0 alive)

Since one node is still up and running and the key space is replicated, I was expecting the queries to succeed. In fact, I'm able to login into the cqlsh on the running node and manually set "consistency one" on the CLI, and execute queries successfully from cqlsh.

But from my Java client, the queries all fail, even though one node is still healthy. But if I use nodetool to remove the down node manually ( nodetool removenode ), the Java client then works fine. i'm using the DataStax Java driver.

Here is a test Java code:

public class CassandraTest {

    public static void main(String[] args) {

            Cluster cluster;
            Session session;
            ResultSet results;
            Row rows;

            // Connect to the cluster and keyspace "demo"
            cluster = Cluster
                            .builder()
                            .addContactPoint("172.31.2.11")
                        //  .withRetryPolicy(DefaultRetryPolicy.INSTANCE)
                        //  .withLoadBalancingPolicy(
                        //                  new TokenAwarePolicy(new DCAwareRoundRobinPolicy()))
                         .build();
                session = cluster.connect("user_data");

                // Insert one record into the users table
                PreparedStatement statement = session.prepare(

                "INSERT INTO user_profile" + "(last_name, user_id, user_roles, email, first_name)"
                                + "VALUES (?,?,?,?,?);");

                BatchStatement batch  = new BatchStatement();
                batch.add (statement.bind("Jones", "22321", "Test Role",
                                "bob@example.com", "Bob"));
                batch.add (statement.bind("Jones2", "222321", "2Test Role",
                                "2bob@example.com", "2Bob"));

                session.execute (batch);
      }      
}

It looks like there are issue with BatchStatements from the DataStax Java driver when one of the Cassandra nodes fail. If I change the code to use BoundStatement, instead of BatchStatement, the Java code works.

Any suggested workarounds to get BatchStatements to work correctly when one of the nodes are down?

Answer 1

For those coming across this post, the mutations (write statements) in a CQL batch need to be persisted to the system.batchlog of TWO nodes to ensure that if a batch fails to get written on a replica, the batch in the batchlog can be replayed to the replica that failed.

This behaviour is the LOGGED part of the LOGGED BATCH . It is a fail-safe that ensures either (a) ALL of the batch statements are applied, or (b) NONE of the batch succeeds because (c) there is no rollback mechanism for failed batches.

In this scenario where there are only two nodes but one node is down, the coordinator cannot persist the batch on two nodes so the batch is marked as failed -- NONE of the statements in the batch are even attempted so there is no rollback required. Cheers!