How to create a data stream in Elasticsearch

Creating a data stream in Elasticsearch provides a stable write target for time-series data while keeping storage organized as the dataset grows. Logs, metrics, and event-style documents stay searchable under one name even as indices roll over behind the scenes.

A data stream is backed by one or more hidden backing indices that share the same mappings and settings from a composable index template. Writes go to the current write index, while searches span all backing indices transparently. Retention and rollover are typically managed by lifecycle features, allowing old data to be removed without changing the stream name used by ingest pipelines.

A matching composable index template is required and must include a data_stream definition plus a @timestamp field mapped as a date. Index requests missing @timestamp are rejected, and a pre-existing index or alias with the same name prevents data stream creation. Security-enabled clusters often require HTTPS and authentication for API calls, so the base URL and curl options may need adjustment.

1. Create a composable index template that enables a data stream for the target name.

   $ curl -s -H "Content-Type: application/json" -X PUT "http://localhost:9200/_index_template/app-logs?pretty" -d '{
     "index_patterns": ["app-logs"],
     "data_stream": {},
     "template": {
       "mappings": {
         "properties": {
           "@timestamp": { "type": "date" },
           "message": { "type": "text" }
         }
       }
     }
   }'
   {
     "acknowledged" : true
   }

   The data stream name must match at least one entry in index_patterns for template selection.

2. Create the data stream.

   $ curl -s -X PUT "http://localhost:9200/_data_stream/app-logs?pretty"
   {
     "acknowledged" : true
   }

   Creation fails if an index, alias, or data stream named app-logs already exists.

3. Index a test document into the data stream with an @timestamp field.

   $ curl -s -H "Content-Type: application/json" -X POST "http://localhost:9200/app-logs/_doc?pretty&refresh=wait_for&filter_path=_index,_id,result" -d '{
     "@timestamp": "2026-01-02T10:35:00Z",
     "message": "login ok"
   }'
   {
     "_index" : ".ds-app-logs-2026.01.06-000001",
     "_id" : "uzXCk5sBM9r8KKMa-je-",
     "result" : "created"
   }

   Documents without @timestamp are rejected by data stream ingestion.

4. Review the data stream template, generation, and backing index name.

   $ curl -s "http://localhost:9200/_data_stream/app-logs?pretty&filter_path=data_streams.name,data_streams.template,data_streams.generation,data_streams.timestamp_field.name,data_streams.indices.index_name"
   {
     "data_streams" : [
       {
         "name" : "app-logs",
         "timestamp_field" : {
           "name" : "@timestamp"
         },
         "indices" : [
           {
             "index_name" : ".ds-app-logs-2026.01.06-000001"
           }
         ],
         "generation" : 1,
         "template" : "app-logs"
       }
     ]
   }

   Backing indices follow names like .ds-app-logs-2026.01.06-000001 and change as rollovers occur.

5. Search the data stream to confirm the indexed document is retrievable.

   $ curl -s "http://localhost:9200/app-logs/_search?pretty&size=1&sort=@timestamp:desc&filter_path=hits.hits._index,hits.hits._id,hits.hits._source"
   {
     "hits" : {
       "hits" : [
         {
           "_index" : ".ds-app-logs-2026.01.06-000001",
           "_id" : "uzXCk5sBM9r8KKMa-je-",
           "_source" : {
             "@timestamp" : "2026-01-02T10:35:00Z",
             "message" : "login ok"
           }
         }
       ]
     }
   }