How to use an Elasticsearch API key with Filebeat output

API keys let Filebeat publish to Elasticsearch without reusing an account password in /etc/filebeat/filebeat.yml. A dedicated key per Filebeat host or environment limits blast radius, simplifies rotation, and keeps writer privileges scoped to the exact log destination.

The output.elasticsearch.api_key setting authenticates Filebeat to the Elasticsearch HTTP API with a single id:api_key value. Current Elastic documentation still expects the raw id and api_key joined by a colon, while the one-time API response also includes an encoded helper value meant for direct Authorization: ApiKey headers instead of the Filebeat setting.

A publishing key only covers normal event delivery. Current Filebeat documentation still recommends loading templates, ILM assets, and module pipelines separately with a more privileged setup credential, then running Filebeat with a lower-privilege writer key. Keep the key in the Filebeat keystore instead of cleartext YAML, add ssl.certificate_authorities or another trusted TLS setting when the cluster uses HTTPS, and include read_pipeline only when enabled modules rely on ingest pipelines.

1. Create an API key scoped to Filebeat publishing privileges.

   $ curl --silent --show-error --fail \
     --user elastic:password \
     --header "Content-Type: application/json" \
     --request POST "https://node-01-secure:9200/_security/api_key?pretty" \
     --data '{
     "name": "filebeat-writer-host",
     "role_descriptors": {
       "filebeat_writer": {
         "cluster": ["monitor"],
         "index": [
           {
             "names": ["filebeat-*"],
             "privileges": ["create_doc", "auto_configure"]
           }
         ]
       }
     }
   }'
   {
     "id" : "Vh3kPZkB9Yj7m1Q2Lx4R",
     "name" : "filebeat-writer-host",
     "api_key" : "o0uH8Q72RiqP2a1Xv8cD7w",
     "encoded" : "Vmgza1Baa0I5WWo3bTFRMkx4NFI6bzB1SDhRNzJSaXFQMmExWHY4Y0Q3dw=="
   }

   Use the id and api_key fields as a single id:api_key value for Filebeat. The returned encoded value is for direct Authorization: ApiKey headers, not for output.elasticsearch.api_key.

   Add expiration when short-lived publishing keys are preferred. If enabled modules depend on ingest pipelines, add the read_pipeline cluster privilege before switching Filebeat to the lower-privilege key.

   If Filebeat setup has not already loaded templates, ILM assets, or pipelines, complete that first with a broader setup credential before using a write-only publishing key.

   Related: How to run Filebeat setup for templates and dashboards
   Related: How to create an Elasticsearch API key

2. Create the Filebeat keystore if this host does not already use one.

   $ sudo filebeat keystore create
   Created filebeat keystore

   Skip this step when /var/lib/filebeat/filebeat.keystore or the deployment's configured path.data keystore already exists.

   Related: How to create a Filebeat keystore

3. Store the id:api_key value in the Filebeat keystore.

   $ printf 'Vh3kPZkB9Yj7m1Q2Lx4R:o0uH8Q72RiqP2a1Xv8cD7w' | sudo filebeat keystore add FILEBEAT_API_KEY --stdin
   Successfully updated the keystore

   Use --stdin to avoid leaving the secret in shell history, and target the same path.data used by the running Filebeat service.

   Related: How to add a secret to a Filebeat keystore

4. Configure the Elasticsearch output to use the keystore-backed API key.

   setup.ilm.check_exists: false
   
   output.elasticsearch:
     hosts: ["https://node-01-secure:9200"]
     api_key: "${FILEBEAT_API_KEY}"
     ssl.certificate_authorities: ["/etc/filebeat/certs/elastic-ca.crt"]

   Keep the placeholder quoted because the resolved secret contains a colon. Set setup.ilm.check_exists: false when templates and ILM assets are already loaded and the publishing key should stay minimal.

   Add ssl.certificate_authorities only when the Elasticsearch CA is not already trusted by the host or container running Filebeat.

5. Test the Filebeat configuration file for syntax errors.

   $ sudo filebeat test config -c /etc/filebeat/filebeat.yml
   Config OK

   Related: How to test a Filebeat configuration

6. Test the Elasticsearch output connection using the stored API key.

   $ sudo filebeat test output -c /etc/filebeat/filebeat.yml
   elasticsearch: https://node-01-secure:9200...
     parse url... OK
     connection...
       parse host... OK
       dns lookup... OK
       addresses: 172.25.0.3
       dial up... OK
     TLS...
       security: server's certificate chain verification is enabled
       handshake... OK
       TLS version: TLSv1.3
       dial up... OK
     talk to server... OK
     version: 9.3.2

   filebeat test output confirms endpoint reachability, TLS, and authentication. It does not publish an event.

   Related: How to test Filebeat output connectivity

7. Restart the Filebeat service to apply the updated output settings.

   $ sudo systemctl restart filebeat

   Related: How to manage the Filebeat service with systemctl in Linux

8. Verify Filebeat is writing to the expected indices or data streams.

   $ curl --silent --show-error --fail \
     --user elastic:password \
     --cacert /etc/filebeat/certs/elastic-ca.crt \
     "https://node-01-secure:9200/_cat/indices/filebeat-*,.ds-filebeat-*?v&expand_wildcards=all"
   health status index                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
   green  open   .ds-filebeat-9.3.2-2026.04.02-000001  3WkHUrIgS_GZcNdCISr52A   1   0         30            0     58.3kb         58.3kb       58.3kb

   Write-only API keys often cannot run verification queries, so use a separate read-capable credential for this check.

   If enabled inputs are idle, wait for the next log line or generate a fresh event before querying Elasticsearch.