MaxScale Galera Disable Master Failback
Introduction
MaxScale galeramon monitor includes an option to prevent that a former Master node recovers its original role whenever it rejoing the Galera Cluster.
Previously we tested the FailBack funtionalities on MaxScale with MariaDB Galera Cluster post,
this time we want to setup the option disable_master_failback and validate the node rejoins with a Slave role.
Master FailBack Disabled
Configure MaxScale Monitor
Edit the MaxScale configuration file:
pato@patomaxgalera:~$ sudo nano /etc/maxscale.cnf
Go to the MaxScale monitor section and add the parameter disable_master_failback=true:
[Pato-Monitor]
type=monitor
module=galeramon
servers=Pato-Node0,Pato-Node1,Pato-Node2
user=maxpato
password=xxxxxxxx
monitor_interval=2000
disable_master_failback=true
Save the file and restart MaxScale for the new configuration to be loaded:
pato@patomaxgalera:~$ sudo systemctl restart maxscale
pato@patomaxgalera:~$
FailBack
Now we are going to test the roles behavior with this new setup when the Master node fails as described:
With this option enabled, if the master status is assigned to a new node it will not be reassigned to the original node for as long as the new master node is running.
Review the current Galera Cluster state issuing maxctrl list servers command:
pato@patomaxscale:~$ maxctrl list servers
+----------------------------------------------------------------------------------+
¦ Server ¦ Address ¦ Port ¦ Connections ¦ State ¦ GTID ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node0 ¦ patomariagm ¦ 3306 ¦ 0 ¦ Master, Synced, Running ¦ 1-1-10 ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node1 ¦ patomariag1 ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-10 ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node2 ¦ patomariag2 ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-10 ¦
+----------------------------------------------------------------------------------+
Stop the designated server patomariagm:
pato@patomariagm:~$ sudo systemctl stop mariadb
and validate the state issuing maxctrl list servers command:
pato@patomaxscale:~$ maxctrl list servers
+----------------------------------------------------------------------------------+
¦ Server ¦ Address ¦ Port ¦ Connections ¦ State ¦ GTID ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node0 ¦ patomariagm ¦ 3306 ¦ 0 ¦ Down ¦ ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node1 ¦ patomariag1 ¦ 3306 ¦ 0 ¦ Master, Synced, Running ¦ 1-1-11 ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node2 ¦ patomariag2 ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-11 ¦
+------------+-------------+------+-------------+-------------------------+--------+
our server is marked as Down and the next server is promoted to Master. We can see the change and switch detected in the MaxScale log:
pato@patomaxscale:~$ tail -f /var/log/maxscale/maxscale.log
2020-06-08 21:43:16 notice : Server changed state: Pato-Node0[patomariagm:3306]: master_down. [Master, Synced, Running] -> [Down]
2020-06-08 21:43:16 notice : Server changed state: Pato-Node1[patomariag1:3306]: new_master. [Slave, Synced, Running] -> [Master, Synced, Running]
2020-06-08 21:43:16 notice : Master switch detected: lost a master and gained a new one
Then if we start the patomariagm server it will not recover its former Master status:
pato@patomariagm:~$ sudo systemctl start mariadb
and validate issuing maxctrl list servers command:
pato@patomaxscale:~$ maxctrl list servers
+-----------------------------------------------------------------------------------------------------+
¦ Server ¦ Address ¦ Port ¦ Connections ¦ State ¦ GTID ¦
+------------+-------------+------+-------------+--------------------------------------------+--------¦
¦ Pato-Node0 ¦ patomariagm ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-12 ¦
+------------+-------------+------+-------------+--------------------------------------------+--------¦
¦ Pato-Node1 ¦ patomariag1 ¦ 3306 ¦ 0 ¦ Master, Synced, Master Stickiness, Running ¦ 1-1-12 ¦
+------------+-------------+------+-------------+--------------------------------------------+--------¦
¦ Pato-Node2 ¦ patomariag2 ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-12 ¦
+-----------------------------------------------------------------------------------------------------+
We can see the change is detected in the MaxScale log:
pato@patomaxscale:~$ tail -f /var/log/maxscale/maxscale.log
2020-06-08 21:43:52 notice : Server changed state: Pato-Node0[patomariagm:3306]: slave_up. [Down] -> [Slave, Synced, Running]
We validated the Master role was not reassigned to the original node
Restore Master Role
In order to restore the Master role to the original server we need to shutdown the current Master:
pato@patomariag1:~$ sudo systemctl stop mariadb
and validate issuing maxctrl list servers command:
pato@patomaxscale:~$ maxctrl list servers
+----------------------------------------------------------------------------------+
¦ Server ¦ Address ¦ Port ¦ Connections ¦ State ¦ GTID ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node0 ¦ patomariagm ¦ 3306 ¦ 0 ¦ Master, Synced, Running ¦ 1-1-14 ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node1 ¦ patomariag1 ¦ 3306 ¦ 0 ¦ Down ¦ ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node2 ¦ patomariag2 ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-14 ¦
+----------------------------------------------------------------------------------+
We can see the change is detected in the MaxScale log:
pato@patomaxscale:~$ tail -f /var/log/maxscale/maxscale.log
2020-06-08 21:46:14 notice : Server changed state: Pato-Node0[patomariagm:3306]: new_master. [Slave, Synced, Running] -> [Master, Synced, Running]
2020-06-08 21:46:14 notice : Server changed state: Pato-Node1[patomariag1:3306]: master_down. [Master, Synced, Master Stickiness, Running] -> [Down]
2020-06-08 21:46:14 notice : Master switch detected: lost a master and gained a new one
And finally we restart the server to return to the original state of the cluster:
pato@patomariag1:~$ sudo systemctl start mariadb
and validate issuing maxctrl list servers command:
pato@patomaxscale:~$ maxctrl list servers
+----------------------------------------------------------------------------------+
¦ Server ¦ Address ¦ Port ¦ Connections ¦ State ¦ GTID ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node0 ¦ patomariagm ¦ 3306 ¦ 0 ¦ Master, Synced, Running ¦ 1-1-15 ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node1 ¦ patomariag1 ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-15 ¦
+------------+-------------+------+-------------+-------------------------+--------¦
¦ Pato-Node2 ¦ patomariag2 ¦ 3306 ¦ 0 ¦ Slave, Synced, Running ¦ 1-1-15 ¦
+----------------------------------------------------------------------------------+
We can see the change is detected in the MaxScale log:
2020-06-08 21:47:04 notice : Server changed state: Pato-Node1[patomariag1:3306]: slave_up. [Down] -> [Slave, Synced, Running]
Conclusion
We configured MaxScale to avoid a failed Master to recover it’s Master role when it recovers and rejoins the Galera Cluster and validated this behavior in our installation for MariaDB Galera Cluster with virtual machine instances on Google Cloud Platform.