Glassfish (Payara) cluster won't start

Question

I'm setting up a cluster for payara 4.1 (last release), i used romote machines for instances and das and when i try to start cluster after intalling, creatin the nodes(1,2) and creating the instances i get this :

start-cluster cluster
remote failure: inst2: Could not start instance inst2 on node node2 (inst2).

Command failed on node node2 (inst2): Previous synchronization failed at Oct 18, 2016 3:03:13 PM
Will perform full synchronization.
Removing all cached state for instance inst2.
NCLS-ADMIN-00010
CLI802 Synchronization failed for directory config, caused by:
  javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateNotYetValidException: NotBefore: Mon Nov 07 09:26:30 CET 2016
Command start-local-instance failed.

To complete this operation run the following command locally on host inst2 from the GlassFish install location /opt/payara41:

 lib/nadmin  start-local-instance --node node2 --sync normal inst2
inst1: Could not start instance inst1 on node node1 (inst1).

Command failed on node node1 (inst1): Previous synchronization failed at Oct 18, 2016 3:03:18 PM
Will perform full synchronization.
Removing all cached state for instance inst1.
NCLS-ADMIN-00010
CLI802 Synchronization failed for directory config, caused by:
  javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateNotYetValidException: NotBefore: Mon Nov 07 09:26:30 CET 2016
Command start-local-instance failed.

To complete this operation run the following command locally on host inst1 from the GlassFish install location /opt/payara41:

 lib/nadmin  start-local-instance --node node1 --sync normal inst1

The command start-instance failed for: inst2 inst1
Command start-cluster failed.

For the cluster :

• cluster name : cluster.
• nodes : node1 , node1.
• instances : inst1, inst2.

I'm using :

• jave 7
• payara 4.1.1.163 #badassfish (build 215)
• Centos 6

Answer 1

The problem is a communications failure between the DAS and the remote node because of an SSL exception due to a certificate validation failure.

The exception is that your certificate is not yet valid which means that your certificate's "valid from" date is in the future. You can also get this if your system time is wrong and in the past (ie a virtual machine you just resumed after many days).

The further issue is that a full synchronization has been attempted. Unfortunately, this works by first deleting all the config, and then trying to retrieve new config from the DAS. In a case such as yours where the DAS is unreachable or communications fail, you are left with a remote node which can't even be started manually.

This behaviour is intended to change in Payara Server eventually but, until then, please avoid using sync=full when starting remote instances.