How to configure Logstash pipelines

Configuring a Logstash pipeline sets the input, filter, and output stages that decide how events enter the service, how fields are changed, and where processed records are sent. A local pipeline file keeps ingest changes reviewable before the service is restarted or reloaded.

Packaged Logstash installs load pipeline definitions from /etc/logstash/pipelines.yml when the service starts without command-line overrides. Each entry names a pipeline.id and a path.config source, and the default main pipeline usually points at /etc/logstash/conf.d/*.conf.

When path.config is a directory or wildcard, Logstash concatenates matching files in lexical order before parsing one effective pipeline. Numeric prefixes such as 10-input.conf, 40-filter.conf, and 90-output.conf keep that order predictable, while centralized pipeline management disables local pipelines.yml, path.config, and config.string settings when it is enabled.

Steps to configure Logstash pipelines:

Read the packaged pipeline manifest.
```
$ sudo cat /etc/logstash/pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
```
If the file lists a different pipeline.id or path.config, save the new pipeline file under that configured location instead of assuming /etc/logstash/conf.d.

If xpack.management.enabled is set to true in /etc/logstash/logstash.yml, local pipeline files are inactive and pipeline changes must be made through centralized pipeline management.

Open the active pipeline file.

$ sudoedit /etc/logstash/conf.d/10-main.conf

Add the pipeline definition.
```
input {
  beats {
    id => "beats_5044"
    port => 5044
  }
}
 
filter {
  mutate {
    add_tag => ["configured_pipeline"]
  }
}
 
output {
  stdout {
    id => "stdout_pipeline_check"
    codec => rubydebug
  }
}
```
The sample keeps one complete pipeline in one file for review. If the same pipeline is split across several files, the filename order decides how input, filter, and output blocks are assembled.

Replace the temporary stdout output with the intended destination after the pipeline starts cleanly. Store destination passwords, API keys, and tokens in the Logstash keystore instead of writing literal secrets into the file.

Related: How to configure a Logstash stdout output
Related: How to configure Logstash output to Elasticsearch
Related: How to add a secret to a Logstash keystore
Test the assembled pipeline configuration.
```
$ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-configtest --config.test_and_exit
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
##### snipped #####
Configuration OK
[2026-06-18T20:36:24,430][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
```
The temporary --path.data directory must be writable by the logstash user and keeps validation state away from the live service data directory under /var/lib/logstash. Current packages default allow_superuser to false, so run package-based tests as the logstash service account unless that setting was intentionally changed.

--config.test_and_exit validates syntax and plugin settings assembly only. It does not prove remote output reachability, credentials, TLS trust, or sender traffic.

Related: How to test a Logstash pipeline configuration

Remove the temporary validation data path.

$ sudo rm --recursive --force /tmp/logstash-configtest

Restart the Logstash service.
```
$ sudo systemctl restart logstash.service
```
Restarting Logstash pauses every active pipeline while inputs reopen, filters recompile, and outputs reconnect.

If /etc/logstash/logstash.yml already sets config.reload.automatic: true, a validated pipeline file change can be picked up without a full service restart. Changes to logstash.yml, plugin installs, JVM options, or service unit overrides still require one.

Related: How to manage the Logstash service with systemctl in Linux
Related: How to enable Logstash pipeline reloads

Confirm that the service returned to the running state.

$ sudo systemctl status logstash.service --no-pager --lines=20
● logstash.service - logstash
     Loaded: loaded (/usr/lib/systemd/system/logstash.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-06-18 20:37:10 UTC; 12s ago
   Main PID: 22164 (java)
      Tasks: 95 (limit: 28486)
     Memory: 962.4M
        CPU: 15.104s
##### snipped #####
Jun 18 20:37:10 logstash-01 systemd[1]: Started logstash.service - logstash.

If the unit does not stay active (running), read the first failure in /var/log/logstash/logstash-plain.log or with sudo journalctl --unit=logstash.service --no-pager --lines=80 before retrying the restart.

Query the monitoring API and confirm that the expected pipeline ID is loaded.

$ curl --silent 'http://localhost:9600/_node/pipelines?pretty'
{
  "host" : "logstash-01",
  "version" : "9.4.2",
  "http_address" : "127.0.0.1:9600",
##### snipped #####
  "pipelines" : {
    "main" : {
      "hash" : "aa78b786a23d82f08d300a3fc07449f2d542b42b976c280bf3bdb9492778dadf",
      "workers" : 8,
      "batch_size" : 125,
      "batch_delay" : 50,
      "batch_output_chunking_growth_threshold_factor" : 1000,
      "dead_letter_queue_enabled" : false
    }
  }
}

If the configured pipeline ID is not main, replace main in later metric or debug requests with the ID returned here.

If this request fails on localhost:9600, check api.enabled, api.http.host, and api.http.port in /etc/logstash/logstash.yml. Current releases enable the API by default, bind it to loopback, and use the 9600-9700 port range.

Review recent startup lines and confirm that the pipeline entered the running state.

$ sudo journalctl --unit=logstash.service --since "5 minutes ago" --no-pager --lines=20
Jun 18 20:37:11 logstash-01 logstash[22164]: [2026-06-18T20:37:11,960][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id" => "main"}
Jun 18 20:37:11 logstash-01 logstash[22164]: [2026-06-18T20:37:11,974][INFO ][logstash.agent           ] Pipelines running {count: 1, running_pipelines: [:main], non_running_pipelines: []}
Jun 18 20:37:12 logstash-01 logstash[22164]: [2026-06-18T20:37:12,018][INFO ][org.logstash.beats.Server][main][beats_5044] Starting server on port: 5044

The journal confirms that Logstash compiled the pipeline, started the beats input listener, and kept the main pipeline running after the service action.

After the pipeline structure is confirmed, replace the temporary stdout output with the real destination output and repeat the configuration test before the next restart or reload.

Author: Mohd Shakir Zakaria
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.