How to use an Elasticsearch ingest pipeline from Logstash
Using an Elasticsearch ingest pipeline from Logstash keeps parsing and light enrichment close to the cluster that finally stores the data. That makes it easier to reuse the same normalization logic across multiple Logstash pipelines, and it gives you one place to confirm that new documents arrive with the fields that dashboards, alerts, and saved searches expect.
The Logstash elasticsearch output sends bulk requests to Elasticsearch, and the output's pipeline setting tells Elasticsearch which ingest pipeline to run before the document is indexed. That lets Elasticsearch processors such as set, rename, date, grok, or geoip modify the event after Logstash has accepted it but before the final document is written.
Current Elastic releases make two details worth checking before you reuse older examples. The pipeline ID is still case-sensitive and must already exist in the destination cluster, while recent Logstash 9.x releases also require current TLS option names such as ssl_enabled and ssl_certificate_authorities and block superuser runs by default unless allow_superuser is explicitly enabled. If you want predictable daily indices for this simple example, keep ILM disabled in the output block instead of mixing rollover aliases or data-stream behavior into the same pipeline.
Steps to use an Elasticsearch ingest pipeline from Logstash:
- Create or update the ingest pipeline in Elasticsearch.
$ curl --silent --show-error --fail \ --cacert /etc/logstash/certs/http_ca.crt \ --user elastic:elastic-password \ -H "Content-Type: application/json" \ -X PUT "https://elasticsearch.example.net:9200/_ingest/pipeline/normalize-logs?filter_path=acknowledged&pretty" \ -d '{ "description": "Mark documents received through Logstash", "processors": [ { "set": { "field": "event.dataset", "value": "logstash.demo" } }, { "set": { "field": "ingest.source", "value": "logstash" } } ] }' { "acknowledged" : true }On self-managed stacks where security is disabled, remove the --cacert and --user options and use http://...:9200// instead of https://...:9200//.
If the pipeline already exists, fetch it first with /_ingest/pipeline/normalize-logs or simulate it before changing processors that other writers may already depend on.
- Reference that pipeline in the Logstash Elasticsearch output.
input { file { path => "/var/lib/logstash/examples/ingest-pipeline.log" start_position => "beginning" sincedb_path => "/var/lib/logstash/sincedb-ingest-pipeline" } } output { elasticsearch { hosts => ["https://elasticsearch.example.net:9200"] ssl_enabled => true ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"] user => "logstash_internal" password => "${LOGSTASH_INTERNAL_PASSWORD}" ilm_enabled => false index => "logstash-ingest-%{+YYYY.MM.dd}" pipeline => "normalize-logs" } }Store the Elasticsearch password in the Logstash keystore or another secret store instead of writing a literal secret in the pipeline file.
The output plugin also supports event-dependent pipeline selection, for example pipeline => "%{[@metadata][pipeline]}". Current plugin docs note that the pipeline parameter is omitted entirely when a dynamic value resolves to an empty string.
Recent 9.x releases removed legacy SSL option names such as ssl and cacert from the Elasticsearch output plugin. Use ssl_enabled and ssl_certificate_authorities instead.
- Test the Logstash pipeline configuration with the packaged settings directory and a temporary data path.
$ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-configtest --config.test_and_exit Using bundled JDK: /usr/share/logstash/jdk Configuration OK [2026-04-07T14:21:08,214][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
The temporary --path.data directory must be writable by the logstash user, and the check validates plugin configuration only. It does not prove that the pipeline ID exists or that Elasticsearch credentials and TLS trust are correct.
Current Logstash 9.x releases reject superuser runs by default, so the packaged test should run as logstash unless allow_superuser is explicitly enabled.
- Restart Logstash or reload the pipeline after the change.
$ sudo systemctl restart logstash
If /etc/logstash/logstash.yml already enables config.reload.automatic, the file change can be picked up without a full service restart. A restart is still the clearest packaged workflow when you want an immediate apply point.
- Generate a fresh event through the configured input and review recent service logs for bulk or pipeline errors.
$ sudo install -o logstash -g logstash -d /var/lib/logstash/examples $ printf '%s\n' 'ingest pipeline example log line' | sudo tee -a /var/lib/logstash/examples/ingest-pipeline.log >/dev/null $ sudo journalctl --unit logstash --since "5 minutes ago" --no-pager Apr 08 08:00:11 host logstash[21457]: [2026-04-08T08:00:11,351][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elasticsearch.example.net:9200/]}} Apr 08 08:00:12 host logstash[21457]: [2026-04-08T08:00:12,884][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} ##### snipped #####If this pipeline does not use the file input shown here, send a fresh event through your real input instead and then read the same journal output for 401, 403, 404, or TLS errors.
A missing ingest pipeline usually shows up as a bulk indexing failure from Elasticsearch, so do not skip the journal check when the final search still returns older documents.
- Query the target index for the newest document and confirm the fields written by the ingest pipeline are present.
$ curl --silent --show-error --fail \ --cacert /etc/logstash/certs/http_ca.crt \ --user reader_user:reader-password \ "https://elasticsearch.example.net:9200/logstash-ingest-*/_search?size=1&sort=@timestamp:desc&_source_includes=message,event.dataset,ingest.source&filter_path=hits.hits._source&pretty" { "hits" : { "hits" : [ { "_source" : { "message" : "ingest pipeline example log line", "event" : { "dataset" : "logstash.demo" }, "ingest" : { "source" : "logstash" } } } ] } }A separate read-capable credential keeps the dedicated Logstash output user focused on write privileges. On unsecured test systems, remove the --cacert and --user options and switch the URL to http://...:9200//.</WRAP>
If the document appears but the event.dataset or ingest.source fields are missing, the event reached the index without the expected ingest pipeline. Re-check the pipeline value in the output block, the journal errors, and the stored pipeline definition in Elasticsearch.