How to resolve GlusterFS split-brain using bigger-file
A file split-brain in a replicated GlusterFS volume is a bigger-file candidate only when the larger replica copy is the version to keep. The resolver tells GlusterFS to copy that largest file over the other conflicting replicas so normal self-heal can converge again.
The server-side gluster volume heal resolver works per conflicted path. First list the split-brain entries, map the volume path to each brick path, then compare the brick-side file sizes before running gluster volume heal <volume> split-brain bigger-file <path> from a node in the trusted pool.
The operation overwrites non-selected replicas, so pause writes to the affected file and keep a backup or copy when the file contents are important. Use another resolver when the files are the same size, when sparse file size does not identify the contents to keep, when timestamps are the better signal, or when the conflict is a type-mismatch directory entry that GlusterFS cannot heal with this policy.
Steps to resolve GlusterFS split-brain using bigger-file:
- Confirm the replica bricks and self-heal daemons are online.
$ sudo gluster volume status volume1 Status of volume: volume1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node1:/srv/gluster/brick1 49152 0 Y 13241 Brick node2:/srv/gluster/brick2 49153 0 Y 13907 Self-heal Daemon on node1 N/A N/A Y 14112 Self-heal Daemon on node2 N/A N/A Y 14203
Bring disconnected bricks back online before resolving split-brain; the selected copy cannot heal to an offline replica.
- List the split-brain entries for the volume.
$ sudo gluster volume heal volume1 info split-brain Brick node1:/srv/gluster/brick1 Number of entries in split-brain: 1 /dir/file1 - Is in split-brain Brick node2:/srv/gluster/brick2 Number of entries in split-brain: 1 /dir/file1 - Is in split-brain
The listed paths are relative to the volume root and should be reused verbatim in the resolve command. Use the gfid: string instead when the output provides only a GFID value.
- Pause application writes to the affected file.
Writes during resolution can create another divergent copy or overwrite data that has not been reviewed.
- Compare the conflicting brick-side file sizes and choose the largest copy deliberately.
$ sudo stat --format='%n %s bytes' /srv/gluster/brick1/dir/file1 /srv/gluster/brick2/dir/file1 /srv/gluster/brick1/dir/file1 8192 bytes /srv/gluster/brick2/dir/file1 4096 bytes
The brick directory from the split-brain listing plus the volume-relative path forms the on-disk path, such as /srv/gluster/brick1/dir/file1.
- Resolve the split-brain entry using the bigger-file policy.
$ sudo gluster volume heal volume1 split-brain bigger-file /dir/file1 Healed /dir/file1.
Run the command once for each path or gfid: value that should use the largest replica copy.
The smaller replica contents are permanently replaced by the larger copy.
- Confirm the split-brain list is empty after the resolution.
$ sudo gluster volume heal volume1 info split-brain Brick node1:/srv/gluster/brick1 Number of entries in split-brain: 0 Brick node2:/srv/gluster/brick2 Number of entries in split-brain: 0
- Check that no self-heal backlog remains for the volume.
$ sudo gluster volume heal volume1 info summary Brick node1:/srv/gluster/brick1 Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick node2:/srv/gluster/brick2 Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0