How to configure Filebeat multiline parsing

Configuring Filebeat multiline parsing keeps stack traces, wrapped exceptions, and continued log records in one event. Without it, one application failure can arrive as dozens of separate documents, making searches, alerts, and dashboards harder to read.

Use the filestream input when adding multiline handling to current Filebeat configurations. Its parsers list can run a multiline parser before processors and outputs, so the event that leaves Filebeat already contains the joined message field.

A good multiline pattern matches the boundary that is most specific for the log format, either the first line of each event or the continuation lines in a stack trace. Keep max_lines and timeout bounded, and assemble multiline events in Filebeat before sending Beats data to Logstash.

1. Find the filestream input that reads the target log path.

   If filebeat.config.inputs.enabled: true loads snippets from /etc/filebeat/inputs.d/*.yml, edit the snippet that owns the path instead of adding a second top-level filebeat.inputs block.

2. Back up the configuration file that contains the target input.

   $ sudo cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak

   Replace /etc/filebeat/filebeat.yml with the matching input snippet path when the host keeps inputs in separate files.

3. Open the target Filebeat configuration file.

   $ sudoedit /etc/filebeat/filebeat.yml

4. Add a multiline parser under the target filestream input.

   filebeat.inputs:
     - type: filestream
       id: app-logs
       enabled: true
       paths:
         - /var/log/app.log
       parsers:
         - multiline:
             type: pattern
             pattern: '^\['
             negate: true
             match: after
             max_lines: 500
             timeout: 5s

   This pattern treats a line starting with [ as the start of a new event and appends following non-matching lines to it, which fits many bracket-prefixed timestamp logs.

   The parsers syntax is for filestream inputs. Deprecated log inputs still use legacy multiline.* keys, so migrate the input instead of mixing both syntaxes.

5. Adjust the pattern for the real log format.

   For Java traces that continue on indented at lines or Caused by: lines, use a continuation pattern such as '^[ \t]+(at|\\.{3})[ \t]+\\b|^Caused by:' with negate: false and match: after.

   Use flush_pattern only when each multiline event has a clear end marker, such as paired Start new event and End event lines.

6. Test the Filebeat configuration.

   $ sudo filebeat test config -c /etc/filebeat/filebeat.yml
   Config OK

   Related: How to test a Filebeat configuration
   Tool: YAML Validator

7. Export the resolved configuration and confirm the parser is attached to the intended input.

   $ sudo filebeat export config -c /etc/filebeat/filebeat.yml
   filebeat:
     inputs:
     - enabled: true
       id: app-logs
       parsers:
       - multiline:
           match: after
           max_lines: 500
           negate: true
           pattern: ^\[
           timeout: 5s
           type: pattern
       paths:
       - /var/log/app.log
       type: filestream
   ##### snipped #####

   The exported output shows the configuration after included files and defaults are resolved, which helps catch indentation mistakes and duplicate input blocks.

8. Restart Filebeat to apply the parser.

   $ sudo systemctl restart filebeat

   Related: How to manage the Filebeat service with systemctl in Linux

9. Check that the Filebeat service is active again.

   $ sudo systemctl is-active filebeat
   active

   Related: How to manage the Filebeat service with systemctl in Linux

10. Send a representative multiline entry through the target log path.

    If a new throwaway test file does not appear immediately, append the sample to an active log or use a file larger than 1024 bytes so the default filestream fingerprint identity can detect it.

11. Confirm that the representative stack trace or wrapped log entry arrives as one event.

    In Elasticsearch or Kibana Discover, the document should contain one message value with embedded newline characters instead of one document per stack frame.
    Related: How to filter data in Kibana Discover

    If unrelated log entries are merged, the pattern is too narrow for event starts or too broad for continuation lines. If stack frames split into separate events, the pattern misses one of the real event headers or continuation formats.