How to skip CSV header lines in Logstash

Skipping CSV headers prevents column titles from being indexed as data, which can pollute dashboards and cause misleading search results when ingesting structured files with Logstash.

The csv filter can treat the first row as the source of column names and discard it when autodetect_column_names and skip_header are enabled. Subsequent rows are parsed into fields using the detected header values as field names.

Header autodetection relies on deterministic event ordering, so the pipeline should run with a single worker while the header row is discovered. Mixed-schema inputs (different headers across files) typically require separate pipelines or explicit columns definitions to avoid incorrect field mapping.

1. Update the csv filter in /etc/logstash/conf.d/20-csv.conf to autodetect columns and skip the header row.

   filter {
     if [log][file][path] == "/var/lib/logstash/input/users.csv" {
       csv {
         autodetect_column_names => true
         skip_header => true
       }
     }
   }

   Header detection uses the first seen CSV row, so the file input must reach the header line (for example when reading from the beginning of a file).

2. Set a single pipeline worker in /etc/logstash/logstash.yml.

   pipeline.workers: 1

   Header skipping is deterministic when a single worker processes events.

   Setting pipeline.workers in /etc/logstash/logstash.yml affects all pipelines on the instance and can reduce throughput.

3. Test the pipeline configuration for errors.

   $ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-configtest --config.test_and_exit
   Configuration OK

   Related: How to test a Logstash pipeline configuration

4. Restart the Logstash service to apply the changes.

   $ sudo systemctl restart logstash

   Restarting Logstash briefly interrupts ingestion while pipelines reload.

   Related: How to manage the Logstash service with systemctl in Linux

5. Check the Logstash service status for a running state.

   $ sudo systemctl --no-pager status logstash
   ● logstash.service - logstash
        Loaded: loaded (/usr/lib/systemd/system/logstash.service; enabled; preset: enabled)
        Active: active (running) since Wed 2026-01-07 12:41:59 UTC; 8s ago
   ##### snipped #####

   Related: How to manage the Logstash service with systemctl in Linux

6. Verify header rows are not indexed in the destination datastore.

   $ curl -s "http://elasticsearch.example.net:9200/users-*/_search?q=name:\"name\"&pretty"
   {
     "took" : 2,
     "timed_out" : false,
     "_shards" : {
       "total" : 1,
       "successful" : 1,
       "skipped" : 0,
       "failed" : 0
     },
     "hits" : {
       "total" : {
         "value" : 0,
         "relation" : "eq"
       },
       "max_score" : null,
       "hits" : [ ]
     }
   }

   Replace users-*, the field name, and the header value to match the pipeline output and CSV header content.

   Header skipping prevents new header events from being indexed, but previously ingested header documents remain until deleted or reindexed.