How to troubleshoot stuck placement groups in Ceph

Stuck placement groups in Ceph mean the cluster cannot finish placing, peering, or cleaning one or more shards of object data. The health warning often names states such as inactive, unclean, undersized, degraded, stale, or peering, and the repair starts by tying that state to a specific placement group and OSD set.

The ceph pg dump_stuck command lists placement groups that have stayed in a problem state beyond the monitor's stuck threshold. Pair it with a PG query, ceph pg map, and the OSD tree so the next action is based on the affected acting set rather than a broad restart or a guess at pool settings.

Start with read-only checks and change only the daemon or host that the PG query identifies. In a cephadm-managed cluster, a confirmed down OSD can be restarted through the orchestrator; if the OSD is down because a disk, host, or network path failed, keep the PG investigation open and move to the matching OSD or host repair path instead of repeatedly restarting the daemon.

1. Read the current Ceph health detail.

   $ ceph health detail
   HEALTH_WARN 1 pgs inactive; 1 pgs stuck inactive; 1 osds down
   [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive
       pg 2.1a is stuck inactive for 36m, current state peering, last acting [1,2,4]
   [WRN] OSD_DOWN: 1 osds down
       osd.2 (root=default,host=ceph-node2) is down

   Record the PG ID, state, and any named OSDs before taking action. A PG can be stuck because it is waiting for peering, missing an acting OSD, blocked by full ratios, or unable to recover cleanly.

2. List the PGs stuck in the state named by the health warning.

   $ ceph pg dump_stuck inactive
   PG_STAT STATE   UP       UP_PRIMARY ACTING   ACTING_PRIMARY
   2.1a    peering [1,2,4]  1          [1,2,4]  1

   Use the state shown by ceph health detail, such as inactive, unclean, stale, undersized, or degraded. Do not add shell filters before reading the native table.

3. Query one affected placement group.

   $ ceph pg 2.1a query
   {
       "state": "peering",
       "up": [1, 2, 4],
       "acting": [1, 2, 4],
       "recovery_state": [
           {
               "name": "Started/Primary/Peering/GetInfo",
               "blocked": "peering is blocked due to down osds",
               "down_osds_we_would_probe": [2]
           }
       ]
   }

   The query output should explain whether peering is blocked by a down OSD, missing acting set, unfound objects, full OSDs, or another condition tied to the PG.

4. Map the placement group to its current acting OSDs.

   $ ceph pg map 2.1a
   osdmap e184 pg 2.1a (2.1a) -> up [1,2,4] acting [1,2,4]

5. Find the host and status of the named OSDs.

   $ ceph osd tree
   ID  CLASS WEIGHT  TYPE NAME           STATUS REWEIGHT PRI-AFF
   -1        0.87354 root default
   -3        0.29118     host ceph-node1
    1    hdd 0.29118         osd.1           up  1.00000 1.00000
   -5        0.29118     host ceph-node2
    2    hdd 0.29118         osd.2         down  1.00000 1.00000
   -7        0.29118     host ceph-node3
    4    hdd 0.29118         osd.4           up  1.00000 1.00000

6. Restart only the down OSD daemon after confirming it should still serve data.

   $ ceph orch daemon restart osd.2
   Scheduled to restart osd.2 on host 'ceph-node2'

   Do not restart unrelated OSDs to clear a stuck PG. If the OSD is down because the disk disappeared, the host is offline, or the daemon repeatedly fails, follow the failed-OSD replacement or host repair path instead.
   Related: How to replace a failed Ceph OSD
   Related: How to manage Ceph services with cephadm

7. Check that recovery or peering has resumed.

   $ ceph -s
     cluster:
       id:     9f0b5b1a-4f2c-4a8d-8e5e-4d0c9f6b7a21
       health: HEALTH_WARN
               Degraded data redundancy: 1 pg recovering
   
     services:
       osd: 3 osds: 3 up, 3 in
   
     data:
       pgs: 1 active+recovering
            255 active+clean

   Recovery can take time on large pools. Watch for progress, not just the absence of the first warning.

8. Confirm that placement groups return to an active clean state.

   $ ceph pg stat
   256 pgs: 256 active+clean; 9.8 TiB data, 29 TiB used, 71 TiB / 100 TiB avail

9. Retest the original health warning.

   $ ceph health detail
   HEALTH_OK

   If the same PG remains stuck, repeat the PG query and follow the new blocker it names, such as unfound objects, full OSD thresholds, or a host that has not rejoined the cluster.