How to configure Elasticsearch disk watermarks

Disk watermarks keep an Elasticsearch node from running out of storage by stopping shard allocation before the filesystem becomes critically full, reducing the risk of write failures and emergency index blocks during rapid growth.

The disk-based shard allocation decider compares each node’s disk usage with the configured low, high, and flood-stage thresholds. Crossing the low watermark prevents new shards from being allocated to the node, crossing the high watermark triggers relocations away from the node, and crossing the flood stage can place affected indices into a read-only mode to preserve disk space.

Watermarks can be expressed as percentages (used disk) or absolute sizes (free disk), and overly aggressive thresholds can keep shards unassigned or prevent rebalancing when the cluster is already near capacity. Apply changes through the cluster settings API with an account that has cluster-level privileges, and ensure enough headroom remains for segment merges, snapshots, and node restarts.

1. Set the low, high, and flood-stage disk watermarks as persistent cluster settings.

   $ curl -s -H "Content-Type: application/json" -X PUT "http://localhost:9200/_cluster/settings" -d '{
     "persistent": {
       "cluster.routing.allocation.disk.watermark.low": "80%",
       "cluster.routing.allocation.disk.watermark.high": "90%",
       "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
     }
   }'
   {
     "acknowledged" : true,
     "persistent" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "disk" : {
               "watermark" : {
                 "low" : "80%",
                 "flood_stage" : "95%",
                 "high" : "90%"
               }
             }
           }
         }
       }
     },
     "transient" : { }
   }

   Percentage values represent used disk space, while byte values such as 50gb represent remaining free space.

2. Verify the configured watermark values from the cluster settings API.

   $ curl -s "http://localhost:9200/_cluster/settings?pretty"
   {
     "persistent" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "disk" : {
               "watermark" : {
                 "low" : "80%",
                 "high" : "90%",
                 "flood_stage" : "95%"
               }
             }
           }
         }
       }
     }
   }

3. List indices that have the read_only_allow_delete block set by flood_stage protection.

   $ curl -s "http://localhost:9200/_all/_settings?filter_path=**.index.blocks.read_only_allow_delete&pretty"
   { }

   Empty output indicates no indices currently have the block set.

4. Clear the read_only_allow_delete block on affected indices after disk usage drops below the flood_stage watermark.

   $ curl -s -H "Content-Type: application/json" -X PUT "http://localhost:9200/_all/_settings" -d '{
     "index.blocks.read_only_allow_delete": null
   }'
   {"acknowledged":true}

   Clearing the block without freeing disk space can cause the block to return and writes to fail when the filesystem fills.

5. Clear custom disk watermark overrides to return to Elasticsearch defaults.

   $ curl -s -H "Content-Type: application/json" -X PUT "http://localhost:9200/_cluster/settings" -d '{
     "persistent": {
       "cluster.routing.allocation.disk.watermark.low": null,
       "cluster.routing.allocation.disk.watermark.high": null,
       "cluster.routing.allocation.disk.watermark.flood_stage": null
     }
   }'
   {
     "acknowledged" : true
   }

6. Confirm the effective default and persistent watermark values after clearing overrides.

   $ curl -s "http://localhost:9200/_cluster/settings?include_defaults=true&filter_path=persistent.cluster.routing.allocation.disk.watermark,defaults.cluster.routing.allocation.disk.watermark.low,defaults.cluster.routing.allocation.disk.watermark.high,defaults.cluster.routing.allocation.disk.watermark.flood_stage&pretty"
   {
     "defaults" : {
       "cluster" : {
         "routing" : {
           "allocation" : {
             "disk" : {
               "watermark" : {
                 "flood_stage" : "95%",
                 "high" : "90%",
                 "low" : "85%"
               }
             }
           }
         }
       }
     }
   }