How to use the Logstash mutate filter
Using the mutate filter in Logstash cleans up field names and values before the event reaches Elasticsearch. That keeps searches, aggregations, alerts, and index mappings more consistent when different senders produce the same data with different casing, extra whitespace, temporary helper fields, or numeric values that still arrive as strings.
The mutate filter performs common field changes such as rename, convert, lowercase, strip, copy, and remove_field during the pipeline filter stage. Current Elastic documentation still defines a fixed processing order inside one mutate block, so later logic that depends on an earlier change should be split into separate mutate blocks instead of assuming the config file order will be honored.
Nested fields must use Logstash field-reference syntax such as [service][name] and [host][name]. Current package installs also block superuser runs by default unless allow_superuser is enabled, so pipeline tests should run as the logstash service account. Changing a field type with convert also does not rewrite an existing Elasticsearch mapping, so a new index or a reindex may still be required if older documents stored the field with the wrong type.
Steps to use the Logstash mutate filter:
- Dry-run the mutate sequence against a sample JSON event before editing the live pipeline.
$ printf '{"service":{"name":" Checkout-API "},"bytes":"1234","environment":"PROD","source_host":"app01"}\n' | sudo -u logstash /usr/share/logstash/bin/logstash \ --path.settings /etc/logstash \ --path.data /tmp/logstash-mutate-dryrun \ -e 'input { stdin { codec => json_lines { ecs_compatibility => disabled } } } filter { mutate { id => "mutate_normalize_fields" strip => ["[service][name]"] lowercase => ["environment"] rename => { "source_host" => "[host][name]" } convert => { "bytes" => "integer" } tag_on_failure => "_mutate_error" } mutate { id => "mutate_enrich_fields" add_field => { "[service][environment]" => "%{environment}" } remove_field => ["environment"] } } output { stdout { codec => rubydebug { metadata => false } } }' { "service" => { "name" => "Checkout-API", "environment" => "prod" }, "host" => { "name" => "app01" }, "bytes" => 1234 }The second mutate block depends on values changed in the first block. Keeping them separate matches the current plugin guidance when the sequence of mutations matters.
- Create the sample file that the service-managed pipeline will watch.
$ sudo install -d -o logstash -g logstash -m 0750 /var/lib/logstash/examples $ sudo install -o logstash -g logstash -m 0640 /dev/null /var/lib/logstash/examples/mutate.log
Starting with an empty file makes the later append step deterministic and avoids replaying older sample lines from a previous test run.
- Add a dedicated pipeline fragment under /etc/logstash/conf.d/60-mutate.conf.
input { file { path => "/var/lib/logstash/examples/mutate.log" start_position => "beginning" sincedb_path => "/var/lib/logstash/sincedb-mutate" codec => json { ecs_compatibility => disabled } tags => ["mutate_demo"] } } filter { if "mutate_demo" in [tags] { mutate { id => "mutate_normalize_fields" strip => ["[service][name]"] lowercase => ["environment"] rename => { "source_host" => "[host][name]" } convert => { "bytes" => "integer" } tag_on_failure => "_mutate_error" } mutate { id => "mutate_enrich_fields" add_field => { "[service][environment]" => "%{environment}" } remove_field => ["environment"] } } } output { if "mutate_demo" in [tags] { elasticsearch { hosts => ["http://elasticsearch.example.net:9200"] ilm_enabled => false index => "app-mutate-%{+YYYY.MM.dd}" } } }The sample input line should look like {"service":{"name":" Checkout-API "},"bytes":"1234","environment":"PROD","source_host":"app01"}. The tags guard keeps this demo filter and output scoped to the sample file instead of every event in the pipeline.
This example intentionally overwrites host.name with the application host carried in source_host. If the file input host metadata and the event origin should both be preserved, rename into another field such as [observer][name] instead.
If the target cluster uses HTTPS or authentication, update the elasticsearch output block to match the real connection settings instead of leaving the plain HTTP placeholder.
- Test the updated pipeline configuration with the packaged settings directory.
$ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-configtest --config.test_and_exit Using bundled JDK: /usr/share/logstash/jdk [2026-04-07T08:25:27,088][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"9.3.2", "jruby.version"=>"jruby 9.4.13.0 (3.1.4) 2025-06-10 9938a3461f OpenJDK 64-Bit Server VM 21.0.10+7-LTS on 21.0.10+7-LTS +indy +jit"} Configuration OK [2026-04-07T08:25:32,961][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting LogstashThe temporary --path.data directory must be writable by the logstash user. This validation proves that the pipeline syntax and plugin settings compile, but it does not prove Elasticsearch connectivity or index privileges.
Current Logstash releases block superuser runs unless allow_superuser is enabled, so run this test as the logstash service account instead of plain root.
- Restart the Logstash service so it loads the updated pipeline.
$ sudo systemctl restart logstash
Restarting Logstash briefly pauses every active pipeline while inputs reopen, filters recompile, and outputs reconnect.
- Append a fresh JSON event to the sample file so the running pipeline has new content to process.
$ printf '{"service":{"name":" Checkout-API "},"bytes":"1234","environment":"PROD","source_host":"app01"}\n' | sudo tee -a /var/lib/logstash/examples/mutate.log >/dev/nullAppending a new line is safer than depending on start_position alone because the dedicated sincedb file remembers what earlier runs already consumed.
- Query the Logstash monitoring API and confirm both named mutate filters handled the event.
$ curl --silent --show-error "http://localhost:9600/_node/stats/pipelines/main?pretty=true&filter_path=pipelines.main.plugins.filters.id,pipelines.main.plugins.filters.name,pipelines.main.plugins.filters.events" { "pipelines" : { "main" : { "plugins" : { "filters" : [ { "id" : "mutate_normalize_fields", "name" : "mutate", "events" : { "in" : 1, "out" : 1 } }, { "id" : "mutate_enrich_fields", "name" : "mutate", "events" : { "in" : 1, "out" : 1 } } ] } } } }If the monitoring API is protected with TLS or basic authentication, adjust the URL and credentials to match the local logstash.yml settings.
- Fetch a recent document from the destination index and confirm the fields were normalized.
$ curl -s -G "http://elasticsearch.example.net:9200/app-mutate-*/_search" \ --data-urlencode "size=1" \ --data-urlencode "sort=@timestamp:desc" \ --data-urlencode "filter_path=hits.hits._index,hits.hits._source.service,hits.hits._source.host,hits.hits._source.bytes" \ --data-urlencode "pretty" { "hits" : { "hits" : [ { "_index" : "app-mutate-2026.04.08", "_source" : { "service" : { "name" : "Checkout-API", "environment" : "prod" }, "host" : { "name" : "app01" }, "bytes" : 1234 } } ] } }If the document still shows bytes as a string or misses the renamed field, an existing index mapping or an earlier filter stage is still overriding the event shape.
- Search the destination index for mutate failure tags after the rollout.
$ curl -s -G "http://elasticsearch.example.net:9200/app-mutate-*/_search" \ --data-urlencode "q=tags:_mutate_error" \ --data-urlencode "size=0" \ --data-urlencode "filter_path=hits.total" \ --data-urlencode "pretty" { "hits" : { "total" : { "value" : 0, "relation" : "eq" } } }A non-zero count means one of the mutate operations failed and the rest of that mutate block stopped for those events. Inspect the event payload and the filter order before retrying.