Parsing log lines with grok in Logstash turns human-written access log text into named event fields. It fits variable formats such as web access logs where a request path, HTTP status, referrer, user agent, and client address need to be searchable as separate values.
The grok filter matches the message field against reusable pattern names and writes named captures back into the event. With ecs_compatibility ⇒ “v8”, composite HTTPD patterns such as HTTPD_COMBINEDLOG produce Elastic Common Schema fields including [source][address], [http][request][method], [http][response][status_code], and [url][original].
Use grok when the text has optional or irregular sections that delimiter parsing cannot handle. Keep the filter in a source-specific branch, give it an id for monitoring, test one representative line before editing the live pipeline, and use the date filter separately when the captured log timestamp should replace @timestamp.
Related: How to use the Logstash dissect filter
Related: How to use the Logstash date filter
Steps to parse logs with grok in Logstash:
- Pick one representative line from the source log and decide the first pattern to test.
For standard Apache-style combined logs, HTTPD_COMBINEDLOG captures the client address, timestamp, request, status, bytes, referrer, and user agent. The Filebeat Grok Pattern Generator can draft a Grok expression from one line, but its ingest-pipeline JSON is not a Logstash pipeline.
Tool: Filebeat Grok Pattern Generator - Create a temporary one-event Grok test pipeline.
input { generator { lines => [ '203.0.113.11 - - [07/Apr/2026:08:17:29 +0000] "GET /grok-demo HTTP/1.1" 200 512 "https://www.example.net/" "Mozilla/5.0"' ] count => 1 } } filter { grok { id => "grok_apache_access" ecs_compatibility => "v8" match => { "message" => "%{HTTPD_COMBINEDLOG}" } tag_on_failure => ["_grokparsefailure_apache"] timeout_scope => "event" } } output { stdout { codec => rubydebug { metadata => false } } }Save the temporary file as /tmp/logstash-grok-dryrun.conf. The generator input emits exactly one sample event, so the run checks the pattern without touching the service pipeline.
- Run the temporary pipeline and confirm the parsed fields are emitted.
$ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-grok-dryrun -f /tmp/logstash-grok-dryrun.conf Using bundled JDK: /usr/share/logstash/jdk ##### snipped ##### { "source" => { "address" => "203.0.113.11" }, "http" => { "request" => { "method" => "GET" }, "response" => { "status_code" => 200, "body" => { "bytes" => 512 } }, "version" => "1.1" }, "url" => { "original" => "/grok-demo" }, "user_agent" => { "original" => "Mozilla/5.0" } }The --config.test_and_exit flag validates pipeline syntax, but it does not check whether a grok pattern matches a real line. This runtime dry run proves the capture names and field nesting.
- Add the grok filter to the source pipeline.
input { file { path => "/var/log/apache2/access.log" start_position => "beginning" sincedb_path => "/var/lib/logstash/sincedb-grok-apache" tags => ["apache_access"] } } filter { if "apache_access" in [tags] { grok { id => "grok_apache_access" ecs_compatibility => "v8" match => { "message" => "%{HTTPD_COMBINEDLOG}" } tag_on_failure => ["_grokparsefailure_apache"] timeout_scope => "event" } } } output { if "apache_access" in [tags] { elasticsearch { hosts => ["http://elasticsearch.example.net:9200"] index => "app-grok-%{+YYYY.MM.dd}" } } }The full fragment shows the input and output boundaries for a package-style pipeline. In an existing pipeline, copy only the grok block into the branch that handles the same log source.
Related: How to configure a Logstash file input
Related: How to configure Logstash output to Elasticsearch - Test the updated pipeline configuration with the packaged settings directory.
$ sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.data /tmp/logstash-configtest --config.test_and_exit Using bundled JDK: /usr/share/logstash/jdk ##### snipped ##### Configuration OK
The temporary --path.data directory keeps validation state away from the live service data directory. This check still does not prove Elasticsearch reachability or new event ingestion.
Related: How to test a Logstash pipeline configuration - Restart logstash.service to load the updated pipeline.
$ sudo systemctl restart logstash.service
Restarting Logstash briefly pauses active ingestion while pipelines stop, compile, and reconnect.
- Confirm the named grok filter is receiving events in the monitoring API.
$ curl --silent --show-error "http://localhost:9600/_node/stats/pipelines/main?pretty=true&filter_path=pipelines.main.plugins.filters.id,pipelines.main.plugins.filters.name,pipelines.main.plugins.filters.events" { "pipelines" : { "main" : { "plugins" : { "filters" : [ { "id" : "grok_apache_access", "name" : "grok", "events" : { "in" : 12844, "out" : 12844 } } ] } } } }Replace main when the filter runs in another pipeline ID. If the API is secured or moved from localhost:9600, use the host, port, TLS, and authentication settings from /etc/logstash/logstash.yml.
Related: How to check Logstash pipeline metrics - Search the destination index for Grok failure tags after the rollout.
$ curl --silent --get "http://elasticsearch.example.net:9200/app-grok-*/_search" \ --data-urlencode "q=tags:_grokparsefailure_apache" \ --data-urlencode "size=0" \ --data-urlencode "filter_path=hits.total" \ --data-urlencode "pretty" { "hits" : { "total" : { "value" : 0, "relation" : "eq" } } }A non-zero count means the pattern no longer matches at least some incoming lines. Split different line shapes with conditionals, tighten the pattern, or switch fixed-format sources to How to use the Logstash dissect filter.
- Fetch a recent document and confirm the parsed fields are present.
$ curl --silent --get "http://elasticsearch.example.net:9200/app-grok-*/_search" \ --data-urlencode "size=1" \ --data-urlencode "sort=@timestamp:desc" \ --data-urlencode "filter_path=hits.hits._index,hits.hits._source.timestamp,hits.hits._source.source.address,hits.hits._source.http.request.method,hits.hits._source.http.version,hits.hits._source.http.response.status_code,hits.hits._source.http.response.body.bytes,hits.hits._source.http.request.referrer,hits.hits._source.url.original,hits.hits._source.user_agent.original" \ --data-urlencode "pretty" { "hits" : { "hits" : [ { "_index" : "app-grok-2026.04.07", "_source" : { "timestamp" : "07/Apr/2026:08:17:29 +0000", "source" : { "address" : "203.0.113.11" }, "http" : { "version" : "1.1", "request" : { "method" : "GET", "referrer" : "https://www.example.net/" }, "response" : { "status_code" : 200, "body" : { "bytes" : 512 } } }, "url" : { "original" : "/grok-demo" }, "user_agent" : { "original" : "Mozilla/5.0" } } } ] } }If the event still arrives with legacy fields such as clientip, verb, request, response, and bytes, the filter is still using legacy pattern captures instead of ECS mode.
- Remove the temporary dry-run and validation paths.
$ sudo rm --recursive --force /tmp/logstash-grok-dryrun /tmp/logstash-configtest /tmp/logstash-grok-dryrun.conf
Keep the saved pipeline fragment under /etc/logstash/conf.d or the configured path.config location.
Mohd Shakir Zakaria is a cloud architect with deep roots in software development and open-source advocacy. Certified in AWS, Red Hat, VMware, ITIL, and Linux, he specializes in designing and managing robust cloud and on-premises infrastructures.
