Logstash pipelines can keep writing new documents long after the original index size and shard layout stop being efficient. Applying an Elasticsearch Index Lifecycle Management (ILM) policy keeps those indices rolling over and aging out on schedule so retention, shard growth, and disk usage stay predictable.
In current Logstash releases, the elasticsearch output can manage the index template and bootstrap the first write index when ILM is enabled. The output writes through a stable rollover alias, installs or rewrites the matching template with index.lifecycle.name plus index.lifecycle.rollover_alias, and lets Elasticsearch move each rolled-over index through the configured lifecycle phases.
This page covers classic alias-based indices, not data streams. Elastic recommends data streams for new append-only time-series pipelines, current Logstash versions can route compatible events into data-stream mode, and ILM is not available in Elasticsearch Serverless, so a custom ILM policy must already exist before startup and data_stream ⇒ false should stay in the output block when the target is a rollover alias instead of a data stream.
Steps to apply an Elasticsearch ILM policy to Logstash indices:
- Confirm the target ILM policy exists in Elasticsearch before changing the Logstash output block.
$ curl --silent --show-error --fail \ --cacert /etc/logstash/certs/http_ca.crt \ --user reader_user:reader-password \ "https://elasticsearch.example.net:9200/_ilm/policy/logstash-hot-warm?pretty" { "logstash-hot-warm" : { "policy" : { "phases" : { "hot" : { "actions" : { "rollover" : { "max_primary_shard_size" : "50gb", "max_age" : "30d" } } }, "delete" : { "min_age" : "90d", "actions" : { "delete" : { } } } } } } }A custom ILM policy must already exist before Logstash starts with ilm_policy ⇒ “logstash-hot-warm”, otherwise the pipeline cannot apply that policy name.
- Update the elasticsearch output in /etc/logstash/conf.d/30-output.conf to enable alias-based ILM for the target index family.
output { elasticsearch { hosts => ["https://elasticsearch.example.net:9200"] ssl_enabled => true ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"] user => "logstash_internal" password => "${LOGSTASH_INTERNAL_PASSWORD}" data_stream => false ilm_enabled => true ilm_policy => "logstash-hot-warm" ilm_rollover_alias => "logstash-ilm" ilm_pattern => "000001" } }Leave manage_template at its default value of true for this workflow so current Logstash installs or rewrites the matching index template and bootstraps the first write index automatically.
Do not combine a dynamic index ⇒ pattern with ilm_rollover_alias. Current Logstash uses the rollover alias as the write target, and dynamic index substitution is not supported with alias-based ILM.
data_stream ⇒ false keeps this output in classic index mode when the pipeline or plugins also support data streams. If the target should be a data stream instead, use the data-stream workflow rather than alias-based ILM on indices.
- Test the pipeline configuration with the packaged settings directory and a temporary data path.
$ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-ilm-configtest --config.test_and_exit Using bundled JDK: /usr/share/logstash/jdk Configuration OK [2026-04-07T14:21:08,214][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
This validation checks pipeline syntax and plugin settings only. It does not prove the Elasticsearch credential, CA file, or policy name are correct.
Current packaged releases default allow_superuser to false, so package-based tests should run as the logstash user unless that setting is deliberately changed.
- Restart the Logstash service so the updated output block loads and the Elasticsearch output can install the template.
$ sudo systemctl restart logstash
Restarting Logstash briefly pauses ingestion while pipelines stop and reconnect.
- Review recent Logstash journal entries and confirm the Elasticsearch output installs the template without startup errors.
$ sudo journalctl --unit logstash --since "5 minutes ago" --no-pager Apr 07 14:24:11 host logstash[21457]: [2026-04-07T14:24:11,351][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elasticsearch.example.net:9200/]}} Apr 07 14:24:12 host logstash[21457]: [2026-04-07T14:24:12,009][INFO ][logstash.outputs.elasticsearch][main] Installing Elasticsearch template {:name=>"logstash-ilm"} Apr 07 14:24:12 host logstash[21457]: [2026-04-07T14:24:12,587][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}If the output user cannot create templates or write to the alias-backed index, the journal usually shows the Elasticsearch error before documents are accepted.
- Check the installed index template and confirm the policy name plus rollover alias match the Logstash output block.
$ curl --silent --show-error --fail \ --cacert /etc/logstash/certs/http_ca.crt \ --user reader_user:reader-password \ "https://elasticsearch.example.net:9200/_index_template/logstash-ilm?pretty" { "index_templates" : [ { "name" : "logstash-ilm", "index_template" : { "index_patterns" : [ "logstash-ilm-*" ], "template" : { "settings" : { "index" : { "lifecycle" : { "name" : "logstash-hot-warm", "rollover_alias" : "logstash-ilm" } } } } } } ] }With template management enabled, current Logstash writes this template automatically. Create it manually only when the output is deliberately running with manage_template ⇒ false.
- Verify the rollover alias resolves to the bootstrapped write index.
$ curl --silent --show-error --fail \ --cacert /etc/logstash/certs/http_ca.crt \ --user reader_user:reader-password \ "https://elasticsearch.example.net:9200/_alias/logstash-ilm?pretty" { "logstash-ilm-000001" : { "aliases" : { "logstash-ilm" : { "is_write_index" : true } } } }If this query returns 404, no document has been indexed through the output yet or template bootstrap failed during pipeline startup.
- Confirm the new index is managed by the intended ILM policy.
$ curl --silent --show-error --fail \ --cacert /etc/logstash/certs/http_ca.crt \ --user reader_user:reader-password \ "https://elasticsearch.example.net:9200/logstash-ilm-000001/_ilm/explain?pretty" { "indices" : { "logstash-ilm-000001" : { "managed" : true, "policy" : "logstash-hot-warm", "phase" : "hot", "action" : "rollover", "step" : "check-rollover-ready" } } }managed: true plus the expected policy name confirms that new indices created through the alias are enrolled in the lifecycle policy. The phase, action, and step values change as rollover and later lifecycle phases run.
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
