Failover testing proves that a clustered MySQL or MariaDB service can survive a node outage without manual intervention. Confirming that the database and its virtual IP relocate cleanly reduces the chance of unexpected downtime during patching, hardware faults, or planned maintenance.
In Pacemaker, high availability for the database is commonly implemented as an active/passive resource group that bundles an IPaddr2 virtual IP with a systemd database service (for example mariadb or mysql). The pcs CLI updates cluster state (such as node standby) and triggers resource movement while respecting ordering and colocation constraints.
Putting a node into standby forces the resource group to move, which can disconnect active sessions and briefly pause writes during the switch. Run the test during a maintenance window, keep at least one alternate node online with quorum, and confirm fencing/quorum behavior to avoid an unintended service outage.
Steps to run a MySQL or MariaDB failover test in Pacemaker:
- Confirm the cluster is online with quorum.
$ sudo pcs status Cluster name: clustername Cluster Summary: * Stack: corosync (Pacemaker is running) * Current DC: node-01 (version 2.1.6-6fdc9deea29) - partition with quorum * Last updated: Thu Jan 1 05:39:19 2026 on node-01 * Last change: Thu Jan 1 05:39:11 2026 by root via cibadmin on node-01 * 3 nodes configured * 2 resource instances configured Node List: * Online: [ node-01 node-02 node-03 ] Full List of Resources: * Resource Group: db-stack: * db_ip (ocf:heartbeat:IPaddr2): Started node-01 * db_service (systemd:mariadb): Started node-01 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabledA partition with quorum and all expected nodes listed as Online indicates the cluster can place resources safely.
- Identify the node currently hosting the database resource group.
$ sudo pcs status resources * Resource Group: db-stack: * db_ip (ocf:heartbeat:IPaddr2): Started node-01 * db_service (systemd:mariadb): Started node-01The node shown as Started for the group resources is the standby target in the next step.
- Place the current database node into standby to trigger failover.
$ sudo pcs node standby node-01
Active client connections through the virtual IP can drop during migration; run during a maintenance window and confirm the alternate node is healthy.
- Confirm the resource group is started on the other node.
$ sudo pcs status resources * Resource Group: db-stack: * db_ip (ocf:heartbeat:IPaddr2): Started node-02 * db_service (systemd:mariadb): Started node-02The virtual IP begins answering on the new host after the IPaddr2 resource shows Started.
- Verify database access through the floating IP.
$ MYSQL_PWD=app-pass mysql -h 192.0.2.20 -u appuser -e "SELECT 1;" 1 1
Using a DNS name mapped to the virtual IP keeps application connection strings stable across failovers.
- Return the original node to active service.
$ sudo pcs node unstandby node-01
Unstandby restores eligibility; the resource group typically stays on the current node unless moved intentionally.
- Confirm the cluster is healthy after the test.
$ sudo pcs status Cluster name: clustername Cluster Summary: * Stack: corosync (Pacemaker is running) * Current DC: node-01 (version 2.1.6-6fdc9deea29) - partition with quorum * Last updated: Thu Jan 1 05:39:26 2026 on node-01 * Last change: Thu Jan 1 05:39:26 2026 by root via cibadmin on node-01 * 3 nodes configured * 2 resource instances configured Node List: * Online: [ node-01 node-02 node-03 ] Full List of Resources: * Resource Group: db-stack: * db_ip (ocf:heartbeat:IPaddr2): Started node-02 * db_service (systemd:mariadb): Started node-02 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabledEnsure there are no Failed Actions and no nodes remain in standby.
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
