When a GlusterFS volume slows down, client timings alone cannot show which brick or file operation is consuming server-side time. Volume profiling adds per-brick counters so an operator can isolate whether LOOKUP, READ, WRITE, FSYNC, or another FOP is carrying latency during a controlled sample window.
The gluster volume profile command enables diagnostic hit counts and latency measurement on the selected volume. After a representative workload runs, profile info reports block-size read/write counters, per-operation latency, call counts, sample duration, and byte totals for each brick.
Volume profiling is a diagnostic switch, not a long-term monitor. Keep the sample short, run it from a trusted pool node, stop profiling after collecting the report, and compare profile output with client-side timing because the report covers the server-side GlusterFS path rather than the full application request path.
Related: How to improve GlusterFS performance
Related: How to monitor GlusterFS health
Steps to profile GlusterFS volume performance:
- Confirm the target volume is started and note the brick names before sampling.
$ sudo gluster volume info volume1 Volume Name: volume1 Type: Distributed-Replicate Volume ID: 9b9f2d4f-3c3a-4c3e-8c10-5b2f8b1f1e32 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: node1:/srv/gluster/brick1 Brick2: node2:/srv/gluster/brick1
Run the profile commands from a node in the trusted storage pool. Use the brick names in later output to spot whether one brick is slower or busier than its peers.
- Start profiling for the volume.
$ sudo gluster volume profile volume1 start Profiling started on volume1
Profiling adds CPU and I/O work on the bricks, so leave it enabled only while collecting the troubleshooting sample.
- Confirm the profiling counters are enabled for the volume.
$ sudo gluster volume info volume1 Volume Name: volume1 Status: Started Options Reconfigured: diagnostics.count-fop-hits: on diagnostics.latency-measurement: on
diagnostics.count-fop-hits records operation counts, and diagnostics.latency-measurement records operation latency while profiling is active.
- Clear old profile counters before starting the workload sample.
$ sudo gluster volume profile volume1 info clear
Clearing counters keeps earlier workload history out of the current incident window.
- Run the workload that reproduces the slow operation from a normal client mount.
Use the same client path, file set, and concurrency that showed the symptom. A synthetic workload can miss directory lookup, small-file, lock, or fsync behavior that appears only in the real application path.
- Display the cumulative profile report after the workload finishes.
$ sudo gluster volume profile volume1 info Brick: node1:/srv/gluster/brick1 ------------------------------------------- Cumulative Stats: Block Size: 1b+ 32b+ 64b+ No. of Reads: 0 2 6 No. of Writes: 184 29 11 Block Size: 128b+ 256b+ 512b+ No. of Reads: 10 18 21 No. of Writes: 37 52 26 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.32 92.10 us 11.00 us 11234.00 us 4124 LOOKUP 1.15 280.55 us 18.00 us 30110.00 us 986 GETXATTR 4.88 1120.44 us 35.00 us 1800970.00 us 211 WRITE 7.41 1698.02 us 29.00 us 2569638.00 us 188 READ 9.07 2110.77 us 44.00 us 7789367.00 us 22 FSYNC Duration : 300 BytesRead : 25165824 BytesWritten : 67108864 ##### snipped #####
Compare Avg-latency, Max-Latency, and No. of calls for the same FOP across bricks. High latency on one brick points toward brick-local disk, filesystem, or process pressure; high call counts across all bricks usually points back to the workload pattern.
- Display an incremental report when comparing the next interval only.
$ sudo gluster volume profile volume1 info incremental Brick: node1:/srv/gluster/brick1 ------------------------------------------- Interval 0 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.44 110.22 us 13.00 us 8120.00 us 911 LOOKUP 3.92 1450.10 us 51.00 us 950120.00 us 47 WRITE 6.87 2012.33 us 38.00 us 1210090.00 us 41 READ Duration : 60 BytesRead : 4194304 BytesWritten : 10485760 ##### snipped #####
Run the incremental command after each short test interval when comparing before/after tuning changes or different client workloads.
- Stop profiling as soon as the sample is collected.
$ sudo gluster volume profile volume1 stop Profiling stopped on volume1
- Verify profiling is disabled for the volume.
$ sudo gluster volume profile volume1 info volume profile: volume1: command failed: Profiling not started
Archive the report with the matching client-side timing, volume type, brick list, and workload description before changing performance options.
Related: How to improve GlusterFS performance
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
