A GlusterFS secondary volume is only useful for recovery when the session that feeds it points in the right direction and can authenticate from every primary node. A geo-replication session records the primary-to-secondary pairing, installs the SSH trust used by gsyncd workers, and leaves a visible Created state before replication is started.
Geo-replication is asynchronous and volume-scoped. The primary cluster sends changes to a secondary target written as user@host::volume, and the secondary user should be confined through gluster-mountbroker rather than granted unrestricted storage access.
Create the session from a primary trusted-pool node after both volumes are started, clocks are synchronized, and a temporary passwordless SSH path exists from that node to one secondary node. The push-pem creation path uses geo-replication public keys generated on the primary cluster and installs session-specific access for all primary nodes; starting the session is a separate operation.
Steps to create a GlusterFS geo-replication session:
- Check the primary volume status from a primary cluster node.
$ sudo gluster volume status gvol-primary Status of volume: gvol-primary Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick pnode1.example.com:/bricks/gvol-primary 49152 0 Y 2149 Brick pnode2.example.com:/bricks/gvol-primary 49153 0 Y 2198 Self-heal Daemon on pnode1.example.com N/A N/A Y 2287 Self-heal Daemon on pnode2.example.com N/A N/A Y 2331
All primary bricks should show Y under Online before the session is created.
Related: How to check GlusterFS volume status - Check the secondary volume status from a secondary cluster node.
$ sudo gluster volume status gvol-secondary Status of volume: gvol-secondary Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------- Brick snode1.example.com:/bricks/gvol-secondary 49152 0 Y 1764 Brick snode2.example.com:/bricks/gvol-secondary 49153 0 Y 1812 Self-heal Daemon on snode1.example.com N/A N/A Y 1901 Self-heal Daemon on snode2.example.com N/A N/A Y 1948
Use an empty secondary volume for the first synchronization so old files do not mix with the replicated copy.
- Create the secondary geo-replication group on each secondary node.
$ sudo groupadd geogroup
Use the same group name on all secondary nodes that host the secondary volume.
- Create the secondary geo-replication account on each secondary node.
$ sudo useradd --create-home --groups geogroup --shell /bin/bash geoaccount
Keep the account name and UID/GID mapping consistent across the secondary cluster when file ownership must remain predictable.
- Configure the mountbroker root and group from one secondary node.
$ sudo gluster-mountbroker setup /var/mountbroker-root geogroup
gluster-mountbroker creates the controlled mount area used by the secondary geo-replication account.
- Allow the secondary volume for the geo-replication account.
$ sudo gluster-mountbroker add gvol-secondary geoaccount
- Check the mountbroker configuration.
$ sudo gluster-mountbroker status
The output should list the mountbroker root, the secondary volume, and the geo-replication account before glusterd is restarted.
- Restart glusterd on all secondary nodes.
$ sudo systemctl restart glusterd
Some distributions use glusterfs-server.service instead of glusterd.service.
Related: How to manage the GlusterFS service with systemctl - Check time synchronization on the primary and secondary nodes.
$ timedatectl show --property=NTPSynchronized --value yes
Clock drift on primary bricks can make geo-replication miss changes during synchronization windows.
- Confirm temporary SSH access from the create node to the main secondary node.
$ ssh geoaccount@snode1.example.com 'hostname --fqdn' snode1.example.com
Use ssh-copy-id from this same primary node if the account still prompts for a password. The push-pem create path needs this temporary login to install geo-replication keys.
- Generate geo-replication SSH keys on the primary cluster.
$ sudo gluster-georep-sshkey generate Generating SSH keys for geo-replication.
The command creates session-specific key material for primary peer nodes and collects the public keys on the node that runs the command.
- Create the geo-replication session with push-pem authentication.
$ sudo gluster volume geo-replication gvol-primary \ geoaccount@snode1.example.com::gvol-secondary \ create push-pem geo-replication: session created
Add ssh-port 50022 before push-pem only when the secondary SSH service listens on a nonstandard port.
Do not reverse the primary and secondary volume names. Geo-replication is directional, and the wrong order points synchronization at the wrong recovery volume.
- Install the geo-replication .pem keys on a secondary node.
$ sudo /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh geoaccount \ gvol-primary gvol-secondary Setting geo-replication PEM keys for geoaccount: done
Some distributions install the helper under /usr/lib/glusterfs with the same script name.
- Verify the new geo-replication session.
$ sudo gluster volume geo-replication gvol-primary \ geoaccount@snode1.example.com::gvol-secondary status PRIMARY NODE PRIMARY VOL PRIMARY BRICK SECONDARY USER SECONDARY SECONDARY NODE STATUS CRAWL STATUS LAST_SYNCED pnode1 gvol-primary /bricks/b1 geoaccount snode1.example.com::gvol-secondary snode1 Created N/A N/A pnode2 gvol-primary /bricks/b2 geoaccount snode1.example.com::gvol-secondary snode2 Created N/A N/A
Created means the session exists and is ready for configuration or start.
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
