How to create a Pacemaker cluster

The first thing to do after installation of cluster components such as Pacemaker, Corosync and pcs is to create a cluster. This then enables us to create and assign resources such as shared IP address or server services such as MySQL to our cluster.

A typical cluster consist of at least 3 nodes, though it is still possible to create a cluster from just 2 nodes. This is to avoid a condition called split-brain where each node in a cluster thinks that it is the only active node and cause unexpected behaviour in a cluster system. The condition is a lot less likely the more nodes there are in a cluster.

A Pacemaker cluster could be created from the terminal using the pcs (Pacemaker/Corosync Configuration System) tool.

1. Launch terminal application on all nodes for the cluster.
2. Install pacemaker, corosync and pcs on all of your cluster nodes.

   How to install Pacemaker, Corosync and pcs on CentOS, Red Hat or Fedora
   How to install Pacemaker, Corosync and pcs on Ubuntu or Debian

3. Open hosts file using your favorite txt editor on all the cluster nodes.

   $ sudo vi /etc/hosts
   [sudo] password for user:

4. Add node's IP and host name information for all cluster nodes to setup a local DNS system.

   192.168.111.11 node-01
   192.168.111.12 node-02

   Cluster nodes are to communicate using assigned name, and it is crucial for each nodes to use static IP address to prevent communication error between the nodes.

5. Test local DNS setup by pinging each nodes from each other using the configured host name.

   $ ping -c3 node-01 && ping -c3 node-01
   PING node-01 (192.168.111.11) 56(84) bytes of data.
   64 bytes from node-01 (192.168.111.11): icmp_seq=1 ttl=64 time=0.061 ms
   64 bytes from node-01 (192.168.111.11): icmp_seq=2 ttl=64 time=0.062 ms
   64 bytes from node-01 (192.168.111.11): icmp_seq=3 ttl=64 time=0.133 ms
   
   --- node-01 ping statistics ---
   3 packets transmitted, 3 received, 0% packet loss, time 2001ms
   rtt min/avg/max/mdev = 0.061/0.085/0.133/0.034 ms
   PING node-01 (192.168.111.11) 56(84) bytes of data.
   64 bytes from node-01 (192.168.111.11): icmp_seq=1 ttl=64 time=0.050 ms
   64 bytes from node-01 (192.168.111.11): icmp_seq=2 ttl=64 time=0.073 ms
   64 bytes from node-01 (192.168.111.11): icmp_seq=3 ttl=64 time=0.070 ms
   
   --- node-01 ping statistics ---
   3 packets transmitted, 3 received, 0% packet loss, time 2002ms
   rtt min/avg/max/mdev = 0.050/0.064/0.073/0.012 ms

6. Authorise all nodes from one of the cluster nodes.

   $ sudo pcs cluster auth node-01 node-02
   Username: hacluster
   Password:
   node-02: Authorized
   node-01: Authorized

7. Create a cluster specifying all the nodes as member from one of the cluster nodes.

   $ sudo pcs cluster setup --name clustername node-01 node-02
   Destroying cluster on nodes: node-01, node-02...
   node-02: Stopping Cluster (pacemaker)...
   node-01: Stopping Cluster (pacemaker)...
   node-01: Successfully destroyed cluster
   node-02: Successfully destroyed cluster
   
   Sending 'pacemaker_remote authkey' to 'node-01', 'node-02'
   node-01: successful distribution of the file 'pacemaker_remote authkey'
   node-02: successful distribution of the file 'pacemaker_remote authkey'
   Sending cluster config files to the nodes...
   node-01: Succeeded
   node-02: Succeeded
   
   Synchronizing pcsd certificates on nodes node-01, node-02...
   node-02: Success
   node-01: Success
   Restarting pcsd on the nodes in order to reload the certificates...
   node-02: Success
   node-01: Success

8. Start cluster for all the cluster nodes from any of the cluster nodes.

   $ sudo pcs cluster start --name clustername --all
   node-01: Starting Cluster (corosync)...
   node-02: Starting Cluster (corosync)...
   node-01: Starting Cluster (pacemaker)...
   node-02: Starting Cluster (pacemaker)...

9. Configure cluster options from any of the cluster nodes.

   $ sudo pcs property set stonith-enabled=false 
   $ sudo pcs property set no-quorum-policy=ignore 

10. Show cluster info to confirm a successful setup from any of the cluster nodes.

    $ sudo pcs cluster status
    Cluster Status:
     Stack: corosync
     Current DC: node-02 (version 1.1.20-5.el7_7.1-3c4c782f70) - partition with quorum
     Last updated: Mon Sep 30 04:40:20 2019
     Last change: Mon Sep 30 04:39:46 2019 by root via cibadmin on node-01
     2 nodes configured
     0 resources configured
    
    PCSD Status:
      node-02: Online
      node-01: Online