How to enable the Logstash dead letter queue

A dead letter queue in Logstash keeps problematic events from being silently dropped when indexing fails, making it possible to investigate data quality issues and replay corrected events later.

When enabled, Logstash writes non-retriable failures into local queue segment files on disk. The dead letter queue is currently used for documents rejected by the Elasticsearch output with HTTP status codes 400 or 404, and for events that trigger errors during conditional statement evaluation.

The queue is stored per pipeline under path.data/dead_letter_queue by default, or under path.dead_letter_queue when overridden. Because storage is local and bounded by configuration, the queue can fill up and start dropping entries, and it still requires operational attention to read, clean, or clear stored segments.

1. Enable the dead letter queue in /etc/logstash/logstash.yml.

   dead_letter_queue.enable: true
   dead_letter_queue.max_bytes: 1024mb
   #dead_letter_queue.storage_policy: drop_newer
   #dead_letter_queue.retain.age: 7d
   #path.dead_letter_queue: /var/lib/logstash/dead_letter_queue

   The dead letter queue is disk-backed and can grow to dead_letter_queue.max_bytes per pipeline, which can exhaust the target filesystem and disrupt ingestion.

2. Test the pipeline configuration for syntax errors.

   $ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-configtest --config.test_and_exit
   Configuration OK

   Related: How to test a Logstash pipeline configuration

3. Restart the Logstash service to activate the queue.

   $ sudo systemctl restart logstash

   Related: How to manage the Logstash service with systemctl in Linux

4. Confirm the dead letter queue metrics are exposed in the pipeline stats API.

   $ curl -s 'http://localhost:9600/_node/stats/pipelines/main?pretty=true'
   {
   ##### snipped #####
         "dead_letter_queue" : {
           "queue_size_in_bytes" : 1,
           "storage_policy" : "drop_newer",
           "max_queue_size_in_bytes" : 1073741824,
           "last_error" : "no errors",
           "expired_events" : 0,
           "dropped_events" : 0
         }
   ##### snipped #####
   }

   The DLQ size is reported at pipelines.<pipeline_id>.dead_letter_queue.queue_size_in_bytes (commonly under the main pipeline).