How to configure GlusterFS server quorum

Network partitions in a GlusterFS trusted pool can leave servers making management decisions from partial cluster membership. Server quorum makes glusterd require the configured percentage of pool members before participating volumes continue serving from that side of the partition.

Two settings are involved. cluster.server-quorum-ratio is a trusted-pool setting and is applied with gluster volume set all, while cluster.server-quorum-type is set on each volume that should participate. When quorum is lost on a node, glusterd can bring down brick processes for participating volumes until enough pool members are reachable again.

Server quorum protects server-side volume configuration and brick process state, not the client I/O path by itself. On a two-node trusted pool, any ratio above 50 percent requires both nodes to remain online, so use a three-node, arbiter, or otherwise documented design when failover during one server outage is required.

Related: How to configure GlusterFS client quorum
Related: How to create a GlusterFS arbiter volume
Related: How to set GlusterFS volume options

Steps to configure GlusterFS server quorum:

  1. Check trusted-pool connectivity from a GlusterFS server. 
    $ sudo gluster peer status
Number of Peers: 2

Hostname: node2
Uuid: 22222222-3333-4444-5555-666666666666
State: Peer in Cluster (Connected)

Hostname: node3
Uuid: 33333333-4444-5555-6666-777777777777
State: Peer in Cluster (Connected)

    Related: How to create a GlusterFS trusted storage pool

  2. Check the target volume before changing quorum participation. 
    $ sudo gluster volume info volume1

Volume Name: volume1
Type: Replicate
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: node1:/bricks/volume1/brick
Brick2: node2:/bricks/volume1/brick
Brick3: node3:/bricks/volume1/brick

    Replace volume1 with the volume that should participate in server-side quorum.

  3. Set the trusted-pool server quorum ratio. 
    $ sudo gluster volume set all cluster.server-quorum-ratio 51%
volume set: success
  4. Enable server quorum participation on the volume. 
    $ sudo gluster volume set volume1 cluster.server-quorum-type server
volume set: success

    On a two-node trusted pool, a ratio above 50 percent means both servers must stay connected. A dummy peer without bricks does not make replica-2 client writes safe from split-brain.

  5. Verify the saved server quorum options. 
    $ sudo gluster volume info volume1

Volume Name: volume1
Type: Replicate
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: node1:/bricks/volume1/brick
Brick2: node2:/bricks/volume1/brick
Brick3: node3:/bricks/volume1/brick
Options Reconfigured:
cluster.server-quorum-type: server
cluster.server-quorum-ratio: 51%
  6. Check brick process state after the change. 
    $ sudo gluster volume status volume1
Status of volume: volume1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/bricks/volume1/brick           49152     0          Y       1844
Brick node2:/bricks/volume1/brick           49153     0          Y       1901
Brick node3:/bricks/volume1/brick           49154     0          Y       1968

Task Status of Volume volume1
------------------------------------------------------------------------------
There are no active volume tasks

    Online: Y on each brick confirms the volume is still serving while quorum is met. Test partition behavior only in a lab or a planned maintenance window.
    Related: How to check GlusterFS volume status