How to skip CSV header rows in Logstash

Skipping CSV header rows in Logstash prevents field names such as name, email, and role from becoming ordinary events in Elasticsearch. This matters when reports, spreadsheet exports, or batch files keep their column labels in the first line, because one indexed header row can distort counts and searches.

The csv filter can use an explicit columns list or learn column names from the first event with autodetect_column_names. Pairing autodetect_column_names with skip_header makes the first row define the fields, and later rows that exactly match that header are dropped instead of sent downstream.

Elastic's plugin reference requires pipeline.workers to be 1 for header skipping, so apply the setting only to the CSV pipeline when possible. A file input that already recorded a sincedb offset will not reread an old header just because start_position is changed, so use a new file or reset that input's state deliberately during testing.

1. Edit the pipeline configuration that parses the CSV rows.

   $ sudoedit /etc/logstash/conf.d/20-csv.conf

2. Set the csv filter to autodetect field names and skip the matching header row.

   filter {
     csv {
       autodetect_column_names => true
       skip_header => true
     }
   }

   Do not set columns in the same filter when autodetect_column_names is enabled. For a fixed schema, use columns ⇒ [“name”, “email”, “role”] with skip_header ⇒ true instead.

   Rows later in the file that exactly match the header values are skipped too.

3. Set pipeline.workers to 1 for the pipeline that uses the header-skipping filter.

   pipeline.workers: 1

   For a dedicated pipeline in /etc/logstash/pipelines.yml, put pipeline.workers: 1 on that pipeline entry instead of lowering every pipeline on the node.

   Lowering workers can reduce throughput for high-volume pipelines, so keep this setting limited to the CSV pipeline when possible.

4. Test the Logstash configuration.

   $ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-configtest --config.test_and_exit
   Using bundled JDK: /usr/share/logstash/jdk
   ##### snipped #####
   Configuration OK
   [2026-06-18T08:03:34,546][INFO ][logstash.runner] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

   Run the test as the logstash user so the temporary data path and packaged settings match the service permission model.
   Related: How to test a Logstash pipeline configuration

5. Restart logstash.service to apply the updated filter and worker setting.

   $ sudo systemctl restart logstash.service

   Restarting Logstash pauses active pipelines while they reload.

   Related: How to manage the Logstash service with systemctl in Linux

6. Check the Logstash service state.

   $ sudo systemctl status logstash.service --no-pager --lines=0
   ● logstash.service - logstash
        Loaded: loaded (/usr/lib/systemd/system/logstash.service; enabled; preset: enabled)
        Active: active (running) since Thu 2026-06-18 08:06:12 UTC; 7s ago
      Main PID: 22164 (java)
         Tasks: 95 (limit: 28486)
        Memory: 962.4M

   Related: How to manage the Logstash service with systemctl in Linux

7. Query the destination index for the header values.

   $ curl --silent --show-error --get http://elasticsearch.example.net:9200/users-*/_count \
     --data-urlencode 'q=name:"name" AND email:"email" AND role:"role"' \
     --data-urlencode pretty
   {
     "count" : 0,
     "_shards" : {
       "total" : 1,
       "successful" : 1,
       "skipped" : 0,
       "failed" : 0
     }
   }

   Replace the index pattern and field names with the destination index and header labels from the CSV being ingested.

   Header skipping affects only events read after the filter and worker settings are applied. Remove older header documents separately if they were already indexed.