How to heal a GlusterFS volume

A GlusterFS volume needs healing after a replica brick returns from an outage, network partition, or maintenance window and some files still show pending self-heal entries. Running the heal commands from a trusted-pool node asks the self-heal daemon to rescan the backlog and gives the operator a count that should return to zero.

Self-heal is meaningful on redundant layouts such as replicated, arbiter-backed, and dispersed volumes. The gluster CLI talks to glusterd, while the self-heal daemon handles the replica work in the background; volume status confirms that bricks and the daemon are online before the heal is triggered.

Healing can copy file data and metadata between bricks, so large backlogs can add disk and network load. Split-brain entries are a separate decision point because GlusterFS cannot choose the authoritative copy automatically, and those entries must be resolved before the heal backlog can fully clear.

Steps to heal a GlusterFS volume:

  1. List volumes to confirm the target volume name.
    $ sudo gluster volume list
    volume1
    volume2
  2. Confirm that the target volume is Started and uses a redundant layout.
    $ sudo gluster volume info volume1
    Volume Name: volume1
    Type: Replicate
    Status: Started
    Number of Bricks: 1 x 2 = 2
    ##### snipped #####

    The gluster volume heal command applies to redundant layouts. A pure distributed volume has no replica or parity copy to heal.

  3. Confirm that bricks and self-heal daemons are online.
    $ sudo gluster volume status volume1
    Status of volume: volume1
    Gluster process                             TCP Port  RDMA Port  Online  Pid
    ------------------------------------------------------------------------------
    Brick node1:/srv/gluster/brick1              49152     0          Y       21435
    Brick node2:/srv/gluster/brick1              49152     0          Y       21987
    Self-heal Daemon on node1                    N/A       N/A        Y       21310
    Self-heal Daemon on node2                    N/A       N/A        Y       21876
    ##### snipped #####
  4. Check for split-brain entries before starting the heal.
    $ sudo gluster volume heal volume1 info split-brain
    Brick node1:/srv/gluster/brick1
    Number of entries in split-brain: 0
    
    Brick node2:/srv/gluster/brick1
    Number of entries in split-brain: 0

    If any entry is listed as split-brain, resolve that entry first instead of forcing a broad heal scan.

  5. Start a self-heal scan for entries that need repair.
    $ sudo gluster volume heal volume1
    Heal operation on volume volume1 has been successful

    Use sudo gluster volume heal volume1 full only when the whole volume needs a complete rescan, because it can walk far more files and increase disk and network load.

  6. Review the pending heal summary.
    $ sudo gluster volume heal volume1 info summary
    Brick node1:/srv/gluster/brick1
    Status: Connected
    Number of entries: 12
    
    Brick node2:/srv/gluster/brick1
    Status: Connected
    Number of entries: 12

    Use sudo gluster volume heal volume1 statistics heal-count when a large volume needs a faster count without a full path listing.

  7. List individual paths when the summary stays nonzero.
    $ sudo gluster volume heal volume1 info
    Brick node1:/srv/gluster/brick1
    /dir1/file-a.log
    /dir2/file-b.db
    
    Brick node2:/srv/gluster/brick1
    /dir1/file-a.log
    /dir2/file-b.db

    Entries marked Is possibly undergoing heal may be locked by an active self-heal pass. Entries marked Is in split-brain need split-brain resolution before they can clear.

  8. Confirm that pending heal entries reach zero.
    $ sudo gluster volume heal volume1 info summary
    Brick node1:/srv/gluster/brick1
    Status: Connected
    Number of entries: 0
    
    Brick node2:/srv/gluster/brick1
    Status: Connected
    Number of entries: 0

    Counts that never reach zero commonly indicate split-brain entries, an offline brick, or a self-heal daemon problem that needs log review.