How to set a quota on a Ceph pool

Ceph pool quotas cap the bytes or object count that one RADOS pool can store. They are useful when a tenant, backup target, test workload, or application pool must stop growing before it crowds other pools on the same cluster.

Ceph stores pool quota values in the OSD map. max_bytes controls stored bytes, max_objects controls object count, and a value of 0 removes that boundary. The quota belongs to the pool, not to a specific RBD image, CephFS path, or RGW bucket.

Choose limits from current pool usage and leave enough headroom for normal writes before applying them to production pools. A pool at or near quota can raise POOL_FULL or POOL_NEAR_FULL health warnings, and writes beyond the boundary fail with No space left on device.

1. Check cluster health before changing the pool.

   $ ceph -s
     cluster:
       id:     11111111-2222-3333-4444-555555555555
       health: HEALTH_OK
   
     services:
       mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3
       mgr: ceph-node1(active), standbys: ceph-node2
       osd: 9 osds: 9 up, 9 in
   
     data:
       pools:   4 pools, 201 pgs
       objects: 284.04k objects, 1.4 TiB
       usage:   4.2 TiB used, 56 TiB / 60 TiB avail
       pgs:     201 active+clean

   Delay quota changes when the cluster already reports degraded, misplaced, backfilling, or full OSDs. A quota can stop client writes immediately after the pool crosses the configured boundary.

   Related: How to check Ceph cluster health

2. Review current usage for the target pool.

   $ ceph df detail
   --- RAW STORAGE ---
   CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
   ssd     60 TiB   56 TiB  4.2 TiB   4.2 TiB       7.00
   
   --- POOLS ---
   POOL              ID   PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
   tenant-archive     7   128  316 GiB   23.78k  948 GiB   8.21    3.6 TiB
   ##### snipped #####

   Set quotas above current STORED and OBJECTS unless the intended outcome is to stop new writes immediately.

3. Set the byte quota for the pool.

   $ ceph osd pool set-quota tenant-archive max_bytes 1099511627776
   set-quota max_bytes = 1099511627776 for pool tenant-archive

   The value 1099511627776 is 1 TiB in bytes. Use an exact byte value when the quota must match a capacity policy.

4. Set the object quota when object count must also be capped.

   $ ceph osd pool set-quota tenant-archive max_objects 1000000
   set-quota max_objects = 1000000 for pool tenant-archive

   Skip max_objects when the pool only needs a byte limit. Set either quota value to 0 later to remove that boundary.

5. Confirm the saved pool quotas.

   $ ceph osd pool get-quota tenant-archive
   quotas for pool 'tenant-archive':
     max objects: 1M objects  (current num objects: 23780 objects)
     max bytes  : 1 TiB  (current num bytes: 316 GiB)

6. Check for quota health warnings after the change.

   $ ceph health detail
   HEALTH_OK

   POOL_NEAR_FULL means the pool is approaching its configured quota. POOL_FULL means the pool has reached or nearly reached quota and can no longer accept writes.

7. Create a local smoke-test object.

   $ truncate -s 1M quota-smoke.bin

8. Write a smoke object only when direct RADOS test objects are allowed in the pool.

   $ rados -p tenant-archive put quota-smoke-001 quota-smoke.bin

   Do not place direct rados objects into an RBD, CephFS, or RGW application pool unless the pool owner has approved that test. Use the application write path instead.

9. Confirm the over-limit failure during a quota boundary test.

   $ rados -p tenant-archive put quota-smoke-over-limit quota-smoke.bin
   error putting quota-smoke-over-limit: (28) No space left on device

   Run an over-limit test only on a scratch pool or during an approved write-stop window. Do not lower a production quota below current usage just to force this error.

10. Remove the smoke object from the pool.

    $ rados -p tenant-archive rm quota-smoke-001 --force-full

    --force-full allows cleanup when the pool or cluster is full enough to block normal operations.

11. Remove the local smoke-test file.

    $ rm quota-smoke.bin