MariaDB/Galera: Is it safe to connect MariaDB 10.4 nodes to a MariaDB 10.1 cluster?

Question

I've got a running MariaDB/Galera cluster with three MariaDB 10.1 nodes and some InnoDB databases. I like to replace the whole cluster by three new nodes. I've already installed MariaDB on the new machines, but using the newer 10.4 version.

Is it safe to just connect the new nodes to the existing cluster to initiate a state transfer and shutdown the old machines afterwards or is this going to fail because of issues between the different versions?

I didn't find anything useful about running clusters with different MariaDB/Galera versions in the docs.

Answer 1

Unfortunately you need to upgrade through each major version of the servers and run mysql_upgrade for each (which is not replicated in the cluster, so that's fine). The risk you run when in a cluster with mixed versions is that a node fails and gets an SST from a higher version, or that something not-backward compatible is done on the higher-versioned node, breaking the lower-versioned node and causing it to request a state-transfer, so you can see that might not end well.

What's more, there are changes in the structure of permission tables between 10.3 and 10.4, band in the Galera protocol, so even if you do not run into the problems mentioned above, replication is likely to fail.

If you can afford it, the safest and easiest way to do a multi-version upgrade is to be sure to configure the node with xtrabackup SSTs for 10.1 and mariabackup SSTs for 10.2 onward, then take the cluster down to a single node, and upgrade that node through 10.2 and 10.3 to 10.4 (running mysql_upgrade each time). Once the node is all the way up to 10.4, then adding more 10.4 nodes is trivial.

This way you only go through the 10.2/mysql_upgrade/10.3/mysql_upgrade_10.4/mysql_upgrade drill on a single node, and you can just directly install 10.4 on the remaining nodes and let them get an SST as they join the cluster.

To make the upgrades easier, here's what you need to do (example for CentOS/RHEL - you can use the same repository script for Debian: the location of config files will be different, package names will be all lowercase, and cleaning up repos will also be a bit different, but that's about it)

1) figure out your current version:

mysqld --version

2) configure the repositories for your current version:

curl -sS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash -s -- --mariadb-server-version=mariadb-10.1.30

3) save your configuration files (these will get overwritten with defaults during the install)

cp /etc/my.cnf.d/server.cnf ~/

4) stop the server

systemctl stop mariadb

5) uninstall the old version

yum remove MariaDB-server MariaDB-client 
# on 10.1 you might also have to remove percona-xtrabackup or percona-xtrabackup-24
# on later versions, this should be replaced with MariaDB-backup
# the rest of the relevant packages are removed as dependencies

6) update the repository definitions

rm -fr /etc/yum.repos.d/mariadb.repo*
rm -fr /var/cache/yum
yum clean metadata
curl -sS https://downloads.mariadb.com/MariaDB/mariadb_repo_setup | sudo bash -s -- --mariadb-server-version=mariadb-10.2

7) install the new server

yum -y install MariaDB-server MariaDB-backup

8) replace the configuration files

cp -pr ~/server.cnf /etc/my.cnf.d/
#you should change the sst method to mariabackup when going from 10.1 to 10.2

9) start the server

galera_new_cluster

10) run mysql_upgrade

mysql_upgrade

11) GOTO step 4) and repeat the process for 10.3 and then 10.4 (remembering that now you're removing/adding MariaDB-backup instead of percona-xtrabackup)

It would probably be easiest to use one of the new nodes to do this by first installing 10.1 on it and letting it join the old cluster, then decommissioning the old cluster and running the upgrade on the new node.

If you can't bring the cluster down to a single node, then you need to upgrade the entire cluster through each major release by doing a rolling upgrade to 10.2, then a rolling upgrade to 10.3, and finally a rolling upgrade to 10.4. Although there is still a risk, it is much more manageable this way. Large 24/7 shops would basically do the rolling upgrade, except they would also reconfigure the list of state transfer donors for each node to only accept transfers from nodes on the same version.

Here's some documentation on using the repository update script: https://mariadb.com/kb/en/mariadb-package-repository-setup-and-usage/