How to configure Filebeat output to Kafka

Publishing Filebeat events to Kafka adds a durable handoff point between log collection and downstream consumers. That buffer helps absorb indexing slowdowns, maintenance windows, and multi-consumer fan-out without forcing every host to wait for the final destination.

Filebeat publishes to Kafka through the output.kafka block in /etc/filebeat/filebeat.yml. The broker list is used to discover cluster metadata, events are JSON-encoded by default, and the target topic can be fixed or built dynamically from event fields such as data_stream.* or a custom field added earlier in the pipeline.

Only one output.* section can stay enabled at a time, so any existing output.elasticsearch or output.logstash block must be commented before Kafka is activated. Current Filebeat releases default the Kafka protocol version to 2.1.0, and Kafka retention or timestamp-validation rules still use the event timestamp from Beats unless the broker or topic is configured for LogAppendTime. Oversized events above max_message_bytes are dropped, and any TLS or SASL settings must match the broker listener that the client uses.

1. Open the Filebeat configuration file.

   $ sudo nano /etc/filebeat/filebeat.yml

   Use the packaged Linux path shown here unless the current deployment uses an archive or container layout with a different config location.

2. Disable other outputs and enable the Kafka output.

   #output.elasticsearch:
   #  hosts: ["https://es.example.net:9200"]
   
   output.kafka:
     hosts: ["kafka-1.example.net:9092", "kafka-2.example.net:9092"]
     topic: "filebeat-logs"
     partition.round_robin:
       reachable_only: true
     required_acks: 1
     compression: gzip

   Filebeat supports only one active output.* block, so comment out any existing output.elasticsearch, output.logstash, output.redis, output.file, or output.console section before enabling output.kafka.

   Use topic for a static destination, or switch to a format string such as '%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}' or '%{[fields.log_topic]}' when Kafka topics should follow event metadata. Current Filebeat releases default the Kafka protocol version to 2.1.0, so an explicit version line is mainly useful when broker compatibility needs to be pinned in the config.

   Events larger than the configured max_message_bytes value are dropped, and TLS or SASL failures will block publishing even when the YAML syntax is valid.

3. Test the configuration for syntax errors.

   $ sudo filebeat test config -c /etc/filebeat/filebeat.yml
   Config OK

   Related: How to test a Filebeat configuration

4. Test the Kafka output connection from the saved configuration.

   $ sudo filebeat test output -c /etc/filebeat/filebeat.yml
   Kafka: kafka-1.example.net:9092...
     parse host... OK
     dns lookup... OK
     addresses: 192.0.2.21
     dial up... OK
   Kafka: kafka-2.example.net:9092...
     parse host... OK
     dns lookup... OK
     addresses: 192.0.2.22
     dial up... OK

   This current 9.3.2 check confirms that the broker address resolves and accepts a TCP connection from the saved output.kafka settings.

   Failures here usually mean the broker hostname, listener port, firewall path, TLS CA chain, or SASL settings do not match the Kafka listener in use.

   Related: How to test Filebeat output connectivity

5. Restart the Filebeat service to apply the Kafka output.

   $ sudo systemctl restart filebeat

   Related: How to manage the Filebeat service with systemctl in Linux

6. Verify the Filebeat service returned to an active state.

   $ sudo systemctl status filebeat --no-pager --lines=15
   ● filebeat.service - Filebeat sends log files to Logstash or Elasticsearch.
        Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; preset: enabled)
        Active: active (running) since Wed 2026-04-02 19:58:41 UTC; 4s ago
   ##### snipped #####

   An active service only confirms that Filebeat started successfully. It does not prove that brokers are accepting the published events.

   Related: How to manage the Filebeat service with systemctl in Linux

7. Review recent Filebeat logs for Kafka connection or publish errors after the restart.

   $ sudo journalctl --unit=filebeat --since "5 min ago" --no-pager | grep -E 'kafka|error|publish'

   Look for connection-established messages, broker addresses, TLS or SASL failures, or repeated backoff and retry lines. Kafka timestamp validation or retention settings can still reject older events after the TCP check succeeds. When delayed events are discarded unexpectedly, review the broker or topic setting for log.message.timestamp.type and switch to LogAppendTime when arrival time should control retention.