How to replace a GlusterFS brick
A GlusterFS brick replacement is a recovery change for a replicated volume when one disk, path, or server entry must leave the layout without changing the client mount point. The risky part is not the command syntax; it is confirming that the replacement brick starts empty and that enough surviving copies remain for healing.
The supported gluster volume replace-brick path for pure replicate and distributed-replicate volumes uses the commit force form. It replaces one HOST:PATH brick entry with another and leaves the self-heal daemon to copy the missing data onto the new brick after the layout changes.
Pure distribute volumes do not have another replica or parity copy to heal from, so replacing a distribute-only brick is an add-brick plus remove-brick migration instead. Schedule the change during a low-traffic window, keep every surviving brick online, and do not replace another brick until the heal backlog returns to zero.
Steps to replace a GlusterFS brick with commit force:
- Display the volume layout and identify the exact brick entry to replace.
$ sudo gluster volume info volume1 Volume Name: volume1 Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: node1:/srv/gluster/brick1 Brick2: node2:/srv/gluster/brick1 Brick3: node1:/srv/gluster/brick2 Brick4: node2:/srv/gluster/brick2
Use this workflow for Replicate or Distributed-Replicate volumes. For a Distribute volume, add the replacement brick and remove the old brick through the remove-brick migration path instead.
- Confirm that the volume has no active task and every surviving brick is online.
$ sudo gluster volume status volume1 Status of volume: volume1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node1:/srv/gluster/brick1 49152 0 Y 1443 Brick node2:/srv/gluster/brick1 N/A N/A N N/A Brick node1:/srv/gluster/brick2 49154 0 Y 1522 Brick node2:/srv/gluster/brick2 49155 0 Y 1531 Self-heal Daemon on node1 N/A N/A Y 1602 Self-heal Daemon on node2 N/A N/A Y 1610 Task Status of Volume volume1 ------------------------------------------------------------------------------ There are no active volume tasks
The brick being replaced can be offline, but the remaining bricks in the protected set must stay online. Stop if another rebalance, remove-brick, replace-brick, or heal-sensitive task is already running.
Related: How to check GlusterFS volume status
- Confirm that the replacement server is connected to the trusted pool.
$ sudo gluster peer status Number of Peers: 2 Hostname: node2 Uuid: 9b947671-67d1-4a46-8e8a-42b2d2b2b3f0 State: Peer in Cluster (Connected) Hostname: node3 Uuid: 3e61c12d-0463-42be-a9e7-5d5875b75d6c State: Peer in Cluster (Connected)
If the replacement server is missing or disconnected, add or repair the trusted-pool peer before using its brick path.
Related: How to create a GlusterFS trusted storage pool - Create the replacement brick directory on the target server.
$ sudo mkdir -p /srv/gluster/brick1
- Confirm that the replacement brick directory is empty.
$ sudo find /srv/gluster/brick1 -mindepth 1 -maxdepth 1 -print
No output means the directory has no visible entries. If the command prints files or directories, stop and choose an empty brick path.
- Confirm that the brick path is on the intended mounted filesystem.
$ sudo df -Th /srv/gluster/brick1 Filesystem Type Size Used Avail Use% Mounted on /dev/sdc1 xfs 2.0T 33G 2.0T 2% /srv/gluster
A brick path on / or a shared system filesystem can fill the operating system volume and disrupt glusterd plus other services.
- Replace the old brick entry with the new brick entry.
$ sudo gluster volume replace-brick volume1 node2:/srv/gluster/brick1 node3:/srv/gluster/brick1 commit force volume replace-brick: success: replace-brick commit successful
Use commit force only after the layout, replacement path, and surviving bricks have been checked. Older start, status, and non-forced commit examples are not the supported replace-brick path for this workflow.
- Confirm that the volume definition lists the new brick.
$ sudo gluster volume info volume1 Volume Name: volume1 Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: node1:/srv/gluster/brick1 Brick2: node3:/srv/gluster/brick1 Brick3: node1:/srv/gluster/brick2 Brick4: node2:/srv/gluster/brick2
- Confirm that the new brick process is online.
$ sudo gluster volume status volume1 Status of volume: volume1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node1:/srv/gluster/brick1 49152 0 Y 1443 Brick node3:/srv/gluster/brick1 49156 0 Y 1834 Brick node1:/srv/gluster/brick2 49154 0 Y 1522 Brick node2:/srv/gluster/brick2 49155 0 Y 1531 Self-heal Daemon on node1 N/A N/A Y 1602 Self-heal Daemon on node2 N/A N/A Y 1610 Self-heal Daemon on node3 N/A N/A Y 1821 Task Status of Volume volume1 ------------------------------------------------------------------------------ There are no active volume tasks
- Check the self-heal backlog until every brick reports zero entries.
$ sudo gluster volume heal volume1 info summary Brick node1:/srv/gluster/brick1 Status: Connected Number of entries: 0 Brick node3:/srv/gluster/brick1 Status: Connected Number of entries: 0 Brick node1:/srv/gluster/brick2 Status: Connected Number of entries: 0 Brick node2:/srv/gluster/brick2 Status: Connected Number of entries: 0
Large replacements can show nonzero entries while the self-heal daemon copies data onto the new brick. Do not retire another brick from the same protected set until the backlog clears.
Related: How to heal a GlusterFS volume