How to prepare a Ceph host for maintenance

A Ceph storage host can carry monitors, managers, OSDs, and gateway or filesystem daemons that clients depend on. Preparing the host for maintenance means proving the cluster can tolerate the outage, stopping the managed daemons through cephadm, and returning them cleanly after the hardware or operating system work is finished.

Cephadm exposes both host maintenance mode and host drain. Maintenance mode stops daemons on one host and records the host as in maintenance, while ceph orch host drain evacuates daemons and schedules OSD removal. Use maintenance mode for a temporary service window; use drain only when data and daemon evacuation are intended.

Start from an administration host with a working cluster config and a keyring allowed to read cluster status and run orchestrator commands. Replace ceph-node2 with the host being serviced, and do not force maintenance if ok-to-stop reports a monitor quorum, manager, or OSD availability problem.

Related: How to remove a host from a Ceph cluster

Steps to prepare a cephadm-managed Ceph host for maintenance:

  1. Check cluster health before changing host state. 
    $ ceph -s
  cluster:
    id:     11111111-2222-3333-4444-555555555555
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 (age 2h)
    mgr: ceph-admin.abc123(active, since 2h)
    osd: 6 osds: 6 up, 6 in

  data:
    pools:   4 pools, 256 pgs
    objects: 3.42M objects, 8.1 TiB
    usage:   24 TiB used, 76 TiB / 100 TiB avail
    pgs:     256 active+clean

    Start with a named health state so host maintenance is not hiding an unrelated recovery, quorum, or placement-group issue.
    Related: How to check Ceph cluster health

  2. List the managed daemons on the host being serviced. 
    $ ceph orch ps --hostname ceph-node2 --refresh
NAME                    HOST        PORTS        STATUS   REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID
mon.ceph-node2          ceph-node2  *:6789,3300  running  5s ago     14d    94.0M        -  20.2.0   1a2b3c4d5e6f  aa11bb22cc33
osd.12                  ceph-node2               running  5s ago     30d     2.0G        -  20.2.0   1a2b3c4d5e6f  bb22cc33dd44
osd.13                  ceph-node2               running  5s ago     30d     2.1G        -  20.2.0   1a2b3c4d5e6f  cc33dd44ee55

    --refresh asks cephadm to update cached daemon state before printing the table.

  3. Check whether Ceph considers the host safe to stop. 
    $ ceph orch host ok-to-stop ceph-node2
It is presumed safe to stop host ceph-node2

    Stop if this command reports that the host is not safe to stop. Forcing maintenance can reduce data availability or break monitor quorum.

  4. Enter maintenance mode for the host. 
    $ ceph orch host maintenance enter ceph-node2
Daemons for Ceph cluster 11111111-2222-3333-4444-555555555555 stopped on host ceph-node2. Host ceph-node2 moved to maintenance mode

    Do not substitute ceph orch host drain for temporary maintenance unless OSD removal is intended. Host drain applies scheduling labels and schedules all OSDs on the host for removal. Use --yes-i-really-mean-it only when the availability risk has been reviewed.

  5. Confirm that cephadm marks the host as in maintenance. 
    $ ceph orch host ls --host-pattern ceph-node2
HOST        ADDR         LABELS  STATUS
ceph-node2  192.0.2.12           Maintenance
1 hosts in cluster whose hostname matched ceph-node2
  6. Complete the hardware, firmware, kernel, or operating system maintenance on the host.

    Keep the host name and network identity unchanged unless a separate host replacement or removal plan is in scope.

  7. Exit maintenance mode after the host is back online. 
    $ ceph orch host maintenance exit ceph-node2
Ceph cluster 11111111-2222-3333-4444-555555555555 on ceph-node2 has exited maintenance mode
  8. Refresh the daemon list for the serviced host. 
    $ ceph orch ps --hostname ceph-node2 --refresh
NAME                    HOST        PORTS        STATUS   REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID
mon.ceph-node2          ceph-node2  *:6789,3300  running  4s ago     14d    94.6M        -  20.2.0   1a2b3c4d5e6f  aa11bb22cc33
osd.12                  ceph-node2               running  4s ago     30d     2.0G        -  20.2.0   1a2b3c4d5e6f  bb22cc33dd44
osd.13                  ceph-node2               running  4s ago     30d     2.1G        -  20.2.0   1a2b3c4d5e6f  cc33dd44ee55

    Any daemon that stays stopped after maintenance should be investigated before returning the host to normal workload rotation.

  9. Check final cluster health after the daemons settle. 
    $ ceph -s
  cluster:
    id:     11111111-2222-3333-4444-555555555555
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-admin,ceph-node1,ceph-node2 (age 3m)
    mgr: ceph-admin.abc123(active, since 2h)
    osd: 6 osds: 6 up, 6 in

  data:
    pools:   4 pools, 256 pgs
    objects: 3.42M objects, 8.1 TiB
    usage:   24 TiB used, 76 TiB / 100 TiB avail
    pgs:     256 active+clean

    If HEALTH_WARN or HEALTH_ERR appears after maintenance, keep the host in the change record and troubleshoot the named health checks before starting the next host.