How to add a new independent IP to a running Hazelcast cluster without restarting existed node?

Question

Hazelcast cluster running in different hosts IP1, IP2... hazelcast.xml configure the TCP-IP members enter image description here

Now I want to expanding the cluster to support more service. I install a new hazelcast in new IP3

How can I add the new IP3 to the exsiting cluster without restarting IP1, IP2?

Answer 1

The members section in TCP is for finding the cluster.

You list some places where cluster members may be. The process starting tries those locations, and if it gets a response the response includes the locations of all cluster members.

When scaling up you frequently won't know the location in advance. The TCP list is one solution, but there's other ways if running on the cloud, etc.

For your specific question: You don't need to add IP3 to your XML. Or you can and it be picked up the next time the processes are restarted.

If you're new to Hazelcast, why not join the community slack

Answer 2

Usually, you don't need to change anything, and just starting a new member (IP3) with listed IP1 and IP2 will work. The third member will join the cluster.

How does it look like under the hood (simplified):

• the newly started member tries to contact addresses in its member list; after a successful connection, it sends a "Who Is Master" request and reads the master node address from the response;
• the new node makes a connection to the master address (if not established already) and asks it to join the cluster;
• if the join is successful, the master replies with an updated member list (cluster view) and it also sends the updated list to all other cluster members;

There were significant improvements in handling corner cases for these "incomplete address configuration" issues in Hazelcast 5.2. So if you are on an older version I strongly recommend switching to an up-to-date one.

If for any reason the default behavior is not sufficient in your case, you can also use the /hazelcast/rest/config/tcp-ip/member-list REST endpoint to make member-list changes. The endpoint was also introduced in 5.2. Find details in the documentation: https://docs.hazelcast.com/hazelcast/5.2.network-partitioning/split-brain-recovery#eliminating-unsuccessful-cluster-merges