How to filter and drop events in Filebeat

Filtering and dropping unwanted events at the Filebeat source saves storage, bandwidth, and processing power downstream. By removing noisy or irrelevant logs, analysis becomes clearer.

Filebeat processors like drop_event or line filtering reduce clutter. Precise filters ensure only valuable data reaches Elasticsearch.

Optimizing event flow at the source leads to leaner, more purposeful data pipelines.

1. Edit filebeat.yml and locate the processors section.

   $ sudo nano /etc/filebeat/filebeat.yml
   (no direct output)

   Processors run after inputs read logs, before sending them out.

2. Add a drop_event processor with conditions to match unwanted logs.
3. Use exclude_lines in inputs to skip logs with certain patterns.

   Regular expressions help precisely target logs for filtering.

4. Test configuration.

   $ sudo filebeat test config
   Config OK

   Incorrect filters may drop important data; test carefully.

5. Restart Filebeat.

   $ sudo systemctl restart filebeat
   (no output)

   Monitor Filebeat output to ensure filtering behaves as expected.