How to rebalance a GlusterFS volume
A GlusterFS volume that has gained bricks can keep old directory layouts that still place new files on the original brick set. Rebalancing the volume updates those layouts and, with the normal start operation, migrates existing files so capacity added to the cluster actually carries data.
The gluster volume rebalance command runs from any server in the trusted storage pool and keeps the volume online while it works. The regular start form fixes layouts and moves data; fix-layout start only refreshes directory layout for future file placement when existing data should not be moved.
Rebalance consumes disk and network I/O, so start it during a lower-traffic window and avoid adding, removing, or replacing bricks until the task finishes. Status output must show each node completed with zero failures before the volume is treated as balanced again.
Steps to rebalance a GlusterFS volume:
- List available volumes to confirm the target name.
$ sudo gluster volume list volume1
- Check the target volume layout and state.
$ sudo gluster volume info volume1 Volume Name: volume1 Type: Distribute Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: node1:/srv/gluster/brick1 Brick2: node2:/srv/gluster/brick1 Brick3: node3:/srv/gluster/brick2
Rebalance is normally run after a topology change such as an added brick. A stopped volume must be started before client-visible placement can be verified.
- Check live brick state before starting the task.
$ sudo gluster volume status volume1 Status of volume: volume1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node1:/srv/gluster/brick1 49152 0 Y 1443 Brick node2:/srv/gluster/brick1 49153 0 Y 1398 Brick node3:/srv/gluster/brick2 49154 0 Y 1522 Task Status of Volume volume1 ------------------------------------------------------------------------------ There are no active volume tasks
Do not start a manual rebalance while another volume task is already active for the same volume.
Related: How to check GlusterFS volume status
- Start the rebalance operation with data migration.
$ sudo gluster volume rebalance volume1 start volume rebalance: volume1: success: Rebalance on volume volume1 has been started successfully
The normal start form migrates existing data. Use gluster volume rebalance volume1 fix-layout start only when existing files should stay in place.
- Check rebalance status while it is running.
$ sudo gluster volume rebalance volume1 status Node Rebalanced-files size scanned failures skipped status run time in sec --------- ---------------- ------ ------- -------- ------- ----------- --------------- node1 416 1.4GB 312 0 0 in progress 74 node2 402 1.3GB 301 0 0 in progress 71 node3 397 1.3GB 298 0 0 in progress 70Re-run the same status command during the maintenance window. The scanned or Rebalanced-files counts should increase between checks until each node leaves in progress.
- Confirm rebalance completed with no failures.
$ sudo gluster volume rebalance volume1 status Node Rebalanced-files size scanned failures skipped status run time in sec --------- ---------------- ------ ------- -------- ------- --------- --------------- node1 1187 10GB 1.1M 0 0 completed 233 node2 1234 10GB 1.2M 0 0 completed 245 node3 1199 10GB 1.1M 0 0 completed 240If failures is nonzero or skipped is higher than expected, inspect the affected brick logs before making another layout change.
Related: How to check GlusterFS logs - Check that the volume has no active tasks after completion.
$ sudo gluster volume status volume1 Status of volume: volume1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick node1:/srv/gluster/brick1 49152 0 Y 1443 Brick node2:/srv/gluster/brick1 49153 0 Y 1398 Brick node3:/srv/gluster/brick2 49154 0 Y 1522 Task Status of Volume volume1 ------------------------------------------------------------------------------ There are no active volume tasks