How to troubleshoot full Ceph OSDs

Capacity warnings in Ceph usually start as OSD_NEARFULL, then OSD_BACKFILLFULL, and finally OSD_FULL when writes are blocked to protect data. The first response is to identify whether one device, one host, or one pool is consuming space unevenly before changing thresholds or moving data.

Ceph reports fullness through the OSD map, so ceph health detail, ceph osd df tree, and ceph df detail should be read together. A pool can have a low-looking stored-data percentage while its MAX AVAIL value is small because the most full OSD inside that pool's CRUSH rule limits placement.

Treat ratio changes as emergency breathing room, not as the repair. Add OSD capacity, delete known-disposable data, rebalance an outlier, or replace a failed OSD according to the signal you find, then retest the original health check.

1. Confirm the exact Ceph fullness health check.

   $ ceph health detail
   HEALTH_ERR 1 full osd(s); 2 backfillfull osd(s); 1 nearfull osd(s)
   [ERR] OSD_FULL: 1 full osd(s)
       osd.7 is full at 96%
   [WRN] OSD_BACKFILLFULL: 2 backfillfull osd(s)
       osd.3 is backfillfull at 91%
       osd.8 is backfillfull at 90%
   [WRN] OSD_NEARFULL: 1 nearfull osd(s)
       osd.5 is nearfull at 87%

   OSD_FULL blocks writes to affected pools. OSD_BACKFILLFULL blocks backfill into the affected OSD. OSD_NEARFULL is the early warning.

2. Identify the full OSD and its host in the CRUSH tree.

   $ ceph osd df tree
   ID  CLASS WEIGHT  REWEIGHT SIZE   RAW USE DATA   OMAP    META  AVAIL %USE  VAR  PGS STATUS TYPE NAME
   -1        54.00000        - 54 TiB 45 TiB 44 TiB 34 GiB 1.2 TiB 8.7 TiB 83.8 1.00   -        root default
   -3        18.00000        - 18 TiB 16 TiB 15 TiB 12 GiB 410 GiB 1.8 TiB 89.9 1.07   -            host ceph-node2
    7   hdd   6.00000  1.00000  6 TiB 5.8 TiB 5.7 TiB 2 GiB 128 GiB 230 GiB 96.2 1.15 124     up         osd.7
    8   hdd   6.00000  1.00000  6 TiB 5.4 TiB 5.3 TiB 8 GiB 140 GiB 590 GiB 90.1 1.08 118     up         osd.8
    9   hdd   6.00000  1.00000  6 TiB 4.8 TiB 4.7 TiB 2 GiB 120 GiB 1.2 TiB 80.3 0.96 109     up         osd.9
   ##### snipped #####

   A high VAR value on one OSD points to imbalance. Similar %USE values across the same device class usually point to real capacity pressure instead.

3. Check the active fullness ratios and OSD map flags.

   $ ceph osd dump
   epoch 772
   fsid 4f6a6b72-1d5a-4d53-ae34-9db8c1d4f0f4
   created 2026-06-02T08:12:31.514522+0000
   modified 2026-06-29T04:36:10.944771+0000
   flags sortbitwise,recovery_deletes
   full_ratio 0.95
   backfillfull_ratio 0.90
   nearfull_ratio 0.85
   ##### snipped #####

   Do not raise full_ratio or backfillfull_ratio as the first fix. A small temporary increase can restore writes in an emergency, but it also reduces the safety margin that prevents OSDs from running out of usable space.

4. Find the pool whose placement is constrained by the full OSD.

   $ ceph df detail
   --- RAW STORAGE ---
   CLASS  SIZE   AVAIL  USED   RAW USED  %RAW USED
   hdd    54 TiB 8.7 TiB 45 TiB 45 TiB       83.80
   TOTAL  54 TiB 8.7 TiB 45 TiB 45 TiB       83.80
   
   --- POOLS ---
   POOL          ID PGS STORED  (DATA)  (OMAP) OBJECTS USED   (DATA)  (OMAP) %USED MAX AVAIL
   rbd            4 256 15 TiB  15 TiB   6 GiB  4.18 M 45 TiB 45 TiB  18 GiB 91.7    510 GiB
   cephfs_data    5 128  3 TiB   3 TiB  28 GiB  1.02 M  9 TiB  9 TiB  84 GiB 34.8    510 GiB
   .mgr           1   1 96 MiB  96 MiB      0 B     31 288 MiB 288 MiB    0 B  0.0    510 GiB

   MAX AVAIL is the important capacity clue when OSD fullness is uneven. If every active pool shows a small MAX AVAIL because of the same OSD, fix that OSD or placement imbalance before treating the whole cluster as evenly full.

5. Check whether recovery or backfill is blocked by space.

   $ ceph -s
   cluster:
     id:     4f6a6b72-1d5a-4d53-ae34-9db8c1d4f0f4
     health: HEALTH_ERR
             1 full osd(s)
             8 pgs backfill_toofull
             Degraded data redundancy: 1245/982344 objects degraded (0.127%), 18 pgs degraded
   
   services:
     mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3
     mgr: ceph-node1(active), standbys: ceph-node2
     osd: 9 osds: 9 up, 9 in
   
   data:
     pools:   3 pools, 385 pgs
     objects: 982.34k objects, 18 TiB
     usage:   45 TiB used, 8.7 TiB / 54 TiB avail
     pgs:     8 active+remapped+backfill_toofull
              18 active+degraded
              359 active+clean

   backfill_toofull means Ceph wants to move data but the target OSD is refusing more backfill because it is already beyond the backfill-full threshold.

   Related: How to troubleshoot stuck placement groups in Ceph

6. Match the finding to the remediation path.

   If one OSD is the outlier and peer OSDs have room, rebalance or reweight that OSD. If all OSDs in the device class are near the same ratio, add capacity or remove known-disposable data from the consuming pool. If a failed OSD or host caused the pressure, replace or recover that component before forcing data movement.

   Related: How to reweight Ceph OSDs by utilization
   Related: How to replace a failed Ceph OSD

7. Recheck the original health detail after the selected remediation.

   $ ceph health detail
   HEALTH_OK

   If the result still reports OSD_BACKFILLFULL or OSD_NEARFULL, repeat the OSD and pool checks before changing thresholds so you know whether the remaining problem is capacity, placement, or a failed component.

8. Confirm that placement groups are no longer blocked by the full OSD.

   $ ceph -s
   cluster:
     id:     4f6a6b72-1d5a-4d53-ae34-9db8c1d4f0f4
     health: HEALTH_OK
   
   services:
     mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3
     mgr: ceph-node1(active), standbys: ceph-node2
     osd: 9 osds: 9 up, 9 in
   
   data:
     pools:   3 pools, 385 pgs
     objects: 982.34k objects, 18 TiB
     usage:   43 TiB used, 11 TiB / 54 TiB avail
     pgs:     385 active+clean