NoHostAvailableException :: Java to Cassandra SSL Cluster connection failed

Question

I am trying to connect to a 3 node Cassandra cluster from a Java client program, with Cassandra cluster configured with Client to node encryption enabled. I deployed three self signed certificate in all three nodes, and imported the public certificates to each of other nodes as per the document Client-to-node encryption . When I run the client program I am getting the following exception.

 Exception in thread "main" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: clm-pun-swpry4/10.133.181.157:9042 (com.datastax.driver.core.exceptions.TransportException: [clm-pun-swpry4/10.133.181.157:9042] Channel has been closed), clm-pun-swpryf/10.133.181.156:9042 (com.datastax.driver.core.exceptions.TransportException: [clm-pun-swpryf/10.133.181.156:9042] Channel has been closed), clm-pun-sqbgda/10.133.172.70:9042 (com.datastax.driver.core.exceptions.TransportException: [clm-pun-sqbgda/10.133.172.70:9042] Channel has been closed))
    at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:233)
    at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:79)
    at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1483)
    at com.datastax.driver.core.Cluster.init(Cluster.java:159)
    at com.datastax.driver.core.SessionManager.initAsync(SessionManager.java:78)
    at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:139)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:68)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:43)
    at clm.bmc.saas.incubator.ClientToNodeExample.main(ClientToNodeExample.java:28)

My Cassandra configuration in cassandra.yaml (individual self signed certificate for each of the 3 nodes):

client_encryption_options:
    enabled: true
    # If enabled and optional is set to true encrypted and unencrypted connections are handled.
    optional: false
    keystore: /opt/secure/keystore.clm-pun-swpry4
    keystore_password: changeit
    # require_client_auth: false
    # Set trustore and truststore_password if require_client_auth is true
    # truststore: conf/.truststore
    # truststore_password: cassandra
    # More advanced defaults below:
    protocol: TLSv1.2
    algorithm: SunX509
    store_type: JKS
    cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA]

My Java program:

public class ClientToNodeExample {
    private static final Logger LOGGER = LoggerFactory.getLogger(ClientToNodeExample.class);

    public static void main(String[] args) {
        ClientToNodeExample example = new ClientToNodeExample();
        Session session = example.getCluster("C:\\install\\ssl\\cassandraCluster.ks", "changeit",
                new String[]{"clm-pun-sqbgda", "clm-pun-swpryf", "clm-pun-swpry4"}, 9042).newSession();
        ResultSet results = session.execute("SELECT * FROM entity_space.mo;");
        LOGGER.info("NumberOfRows:" + results.all().size());
        session.close();
    }

    private Cluster getCluster(String trustStoreLocation, String trustStorePassword, String[] host, int port) {
        Cluster cluster;
        SSLContext sslcontext = null;
        try {
            InputStream is = ClientToNodeExample.class.getResourceAsStream(trustStoreLocation);
            KeyStore keystore = KeyStore.getInstance("jks");
            char[] pwd = trustStorePassword.toCharArray();
            keystore.load(is, pwd);
            TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
            tmf.init(keystore);
            TrustManager[] tm = tmf.getTrustManagers();
            sslcontext = SSLContext.getInstance("TLS");
            sslcontext.init(null, tm, null);
        } catch (Exception e) {
            LOGGER.error("ERROR", e);
        }
        JdkSSLOptions sslOptions = JdkSSLOptions.builder().withSSLContext(sslcontext).build();
        cluster = Cluster.builder().addContactPoints(host).withPort(port).withSSL(sslOptions).build();
        return cluster;
    }
}

However if I try to check the certificate from each node with the following keytool command, I get the certificate:

keytool -printcert -sslserver clm-pun-sqbgda:9042 -rfc

Can anyone please help where I am going wrong?

Cassandra version:3.11.0
cassandra-driver-core : 3.1.4

Answer 1

First I would try to connect via cqlsh --ssl (you need to do some extra steps):

• convert server certificate to PKCS12 format
• convert PKCS12 to PEM
• modify cqlshrc file in order to use the PEM certificate.

If this is ok, then the setup is correct and I would remove the sslOptions and provide

-Djavax.net.ssl.trustStore
-Djavax.net.ssl.trustStorePassword 
-Djavax.net.debug=ssl.

If this is works also, then the issue is in your getCluster method.

My guess is that your keystore (cassandraCluster.ks) is not loaded correctly.

I successfully executed your code as maven project, with the keystore in the resources folder and loaded with

cluster = getCluster("/client-truststore.jks", "changeit", new String[]{"127.0.0.1"}, 9042);