How to check GlusterFS geo-replication logs

GlusterFS geo-replication issues often show up as rising lag, missing updates, or sessions that look healthy while data quietly stops moving. The geo-replication logs are the fastest way to confirm whether the problem is changelog processing, SSH transport, or the sync worker itself.

GlusterFS geo-replication runs gsyncd processes per session, writing local logs on both the primary and the secondary nodes. Primary-side logs are typically stored under /var/log/glusterfs/geo-replication and secondary-side logs under /var/log/glusterfs/geo-replication-secondary, with subdirectories derived from volume and session names.

Log layouts can vary slightly across GlusterFS versions and can include rotated or compressed files, so discovery commands are more reliable than memorized filenames. Geo-replication logs can also grow quickly and may include hostnames, usernames, and file paths, so limit inspection and sharing to the smallest time window that explains the failure.

1. List geo-replication log directories on a primary node.

   $ sudo ls -1 /var/log/glusterfs/geo-replication
   gvol-primary

2. Locate the gsyncd log files for the affected primary volume.

   $ sudo find /var/log/glusterfs/geo-replication/gvol-primary -type f -name 'gsyncd.log*' -print
   /var/log/glusterfs/geo-replication/gvol-primary/gsyncd.log
   /var/log/glusterfs/geo-replication/gvol-primary/gsyncd.log.1
   /var/log/glusterfs/geo-replication/gvol-primary/gsyncd.log.2.gz

3. Inspect the most recent entries from the primary gsyncd log.

   $ sudo tail -n 30 /var/log/glusterfs/geo-replication/gvol-primary/gsyncd.log
   [2025-12-25 10:07:14] I [master] Worker spawned: pid=18423
   [2025-12-25 10:07:14] I [changelog] Processing changelog: CHANGELOG.1703508432
   [2025-12-25 10:07:15] I [ssh] Connecting to secondary: replica1.example.net
   [2025-12-25 10:07:16] I [sync] Sync completed: files=12 bytes=3.1M
   ##### snipped #####

4. Search current and rotated primary logs for common error signatures.

   $ sudo zgrep -n -E 'ERROR|WARNING|Traceback|Permission denied|Host key verification failed|Connection timed out|No route to host|Broken pipe' /var/log/glusterfs/geo-replication/gvol-primary/gsyncd.log* | tail -n 10
   12489:[2025-12-25 09:42:01] E [ssh] Host key verification failed.
   12490:[2025-12-25 09:42:01] E [replica] ssh exited with status 255
   ##### snipped #####

   Related: How to configure the log level for an SSH server

5. List geo-replication log directories on a secondary node.

   $ sudo ls -1 /var/log/glusterfs/geo-replication-secondary
   gvol-secondary

6. Locate the gsyncd log files for the affected secondary volume.

   $ sudo find /var/log/glusterfs/geo-replication-secondary/gvol-secondary -type f -name 'gsyncd.log*' -print
   /var/log/glusterfs/geo-replication-secondary/gvol-secondary/gsyncd.log
   /var/log/glusterfs/geo-replication-secondary/gvol-secondary/gsyncd.log.1.gz

7. Inspect the most recent entries from the secondary gsyncd log.

   $ sudo tail -n 30 /var/log/glusterfs/geo-replication-secondary/gvol-secondary/gsyncd.log
   [2025-12-25 10:07:15] I [replica] Connection received from primary: master1.example.net
   [2025-12-25 10:07:16] I [sync] Applying change set: 12 items
   [2025-12-25 10:07:16] I [replica] Acknowledged checkpoint: 1703508432
   ##### snipped #####

8. Search current and rotated secondary logs for the same error signatures.

   $ sudo zgrep -n -E 'ERROR|WARNING|Traceback|Permission denied|No space left on device|Read-only file system' /var/log/glusterfs/geo-replication-secondary/gvol-secondary/gsyncd.log* | tail -n 10
   ##### snipped #####

9. Follow the primary gsyncd log across rotation to confirm new events are still being written.

   $ sudo tail -F /var/log/glusterfs/geo-replication/gvol-primary/gsyncd.log
   [2025-12-25 10:09:02] I [checkpoint] Checkpoint advanced: 1703508542
   ##### snipped #####

10. Check glusterd log messages when the session directory is missing or the gsyncd log stays idle.

    $ sudo grep -n 'gsyncd' /var/log/glusterfs/glusterd.log | tail -n 5
    4281:[2025-12-25 09:41:55] I [geo-replication] starting gsyncd for session gvol-primary -> gvol-secondary
    4282:[2025-12-25 09:42:01] E [geo-replication] gsyncd exited with status 255

11. Save a small log excerpt for troubleshooting without copying the entire file.

    $ sudo tail -n 2000 /var/log/glusterfs/geo-replication/gvol-primary/gsyncd.log | sudo tee /tmp/gvol-primary-gsyncd.log >/dev/null

    Geo-replication logs can include hostnames, usernames, and file paths, so redact sensitive lines before sharing outside the cluster.