How to optimize Logstash pipeline performance

Optimizing Logstash pipelines improves throughput, reduces end-to-end latency, and prevents ingestion backlogs when event volume spikes.

Pipeline performance is primarily shaped by how many pipeline workers execute filters and outputs in parallel, how efficiently plugins process each event, and how batching amortizes per-event overhead. The internal queue and batch settings determine how events move through the pipeline, while output backpressure can dominate latency when downstream systems slow down.

Tuning is workload-dependent and should be done incrementally with measurements taken before and after each change. Higher concurrency can increase context switching, larger batches can increase memory pressure, bigger heaps can increase garbage-collection pause time, and persistent queues trade disk I/O for buffering and durability. Paths and service commands below assume a package-based Logstash installation on Linux with systemd.

1. Capture baseline pipeline and plugin stats from the Logstash monitoring API.

   $ curl -s 'http://localhost:9600/_node/stats/pipelines?filter_path=pipelines.main.events,pipelines.main.plugins.filters,pipelines.main.plugins.outputs&pretty'
   {
   ##### snipped #####
     "pipelines" : {
       "main" : {
         "events" : {
           "duration_in_millis" : 2248,
           "queue_push_duration_in_millis" : 56,
           "filtered" : 604,
           "out" : 604,
           "in" : 330
         },
         "plugins" : {
           "filters" : [ {
             "events" : {
               "duration_in_millis" : 0,
               "out" : 0,
               "in" : 0
             },
             "name" : "grok"
   ##### snipped #####
           }, {
             "events" : {
               "duration_in_millis" : 8,
               "out" : 0,
               "in" : 0
             },
             "name" : "dissect"
   ##### snipped #####
           } ],
           "outputs" : [ {
             "events" : {
               "duration_in_millis" : 174,
               "out" : 604,
               "in" : 604
             },
             "name" : "elasticsearch"
   ##### snipped #####
           } ]
         }
       }
     }
   }

   Average processing cost can be estimated by dividing duration_in_millis by events.out for the pipeline or a specific plugin.

2. Tune worker and batch settings in /etc/logstash/logstash.yml.

   pipeline.workers: 4
   pipeline.batch.size: 200
   pipeline.batch.delay: 50

   For CPU-bound parsing, start pipeline.workers near the number of available CPU cores; larger pipeline.batch.size can improve throughput at the cost of memory and per-event latency.

3. Enable persistent queues in /etc/logstash/logstash.yml for disk-backed buffering.

   queue.type: persisted
   queue.max_bytes: 4gb

   Persistent queues consume disk space and add disk I/O; if the filesystem hosting path.data fills up, Logstash can stall or stop accepting new events.

4. Replace regex-heavy parsing with dissect where possible in pipeline configuration under /etc/logstash/conf.d.

   filter {
     dissect {
       mapping => { "message" => "%{ts} %{level} %{msg}" }
     }
   }

   Prefer dissect for fixed-delimiter formats; keep grok limited to message types that truly need regular expressions, and gate it with conditionals so it does not run for every event.

5. Increase JVM heap sizes in /etc/logstash/jvm.options.d/heap.options.

   -Xms2g
   -Xmx2g

   Set -Xms and -Xmx to the same value to reduce resizing overhead; edit /etc/logstash/jvm.options when /etc/logstash/jvm.options.d is not used by the installation.

6. Test the pipeline configuration for syntax errors.

   $ sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.test_and_exit
   Config Validation Result: OK. Exiting Logstash

   Related: How to test a Logstash pipeline configuration

7. Restart the Logstash service to apply the updated configuration.

   $ sudo systemctl restart logstash

   Related: How to manage the Logstash service with systemctl in Linux

8. Confirm the Logstash service is active.

   $ sudo systemctl status logstash --no-pager -l
   ● logstash.service - logstash
        Loaded: loaded (/usr/lib/systemd/system/logstash.service; enabled; preset: enabled)
        Active: active (running) since Wed 2026-01-07 23:27:45 UTC; 2min 32s ago
      Main PID: 50336 (java)
         Tasks: 148 (limit: 28486)
        Memory: 1.1G (peak: 1.1G)
           CPU: 53.661s
   ##### snipped #####

9. Capture pipeline and plugin stats again to validate the post-change processing cost.

   $ curl -s 'http://localhost:9600/_node/stats/pipelines?filter_path=pipelines.main.events,pipelines.main.plugins.filters,pipelines.main.plugins.outputs&pretty'
   {
   ##### snipped #####
     "pipelines" : {
       "main" : {
         "events" : {
           "duration_in_millis" : 2514,
           "queue_push_duration_in_millis" : 56,
           "filtered" : 607,
           "out" : 607,
           "in" : 333
         },
         "plugins" : {
           "filters" : [ {
             "events" : {
               "duration_in_millis" : 0,
               "out" : 0,
               "in" : 0
             },
             "name" : "grok"
   ##### snipped #####
           }, {
             "events" : {
               "duration_in_millis" : 9,
               "out" : 0,
               "in" : 0
             },
             "name" : "dissect"
   ##### snipped #####
           } ],
           "outputs" : [ {
             "events" : {
               "duration_in_millis" : 185,
               "out" : 607,
               "in" : 607
             },
             "name" : "elasticsearch"
   ##### snipped #####
           } ]
         }
       }
     }
   }

   For meaningful comparison, collect metrics under similar traffic and compare per-event cost (duration_in_millis divided by events.out) instead of raw counters.