How to bulk index documents into Elasticsearch

Bulk indexing in Elasticsearch leverages the bulk API to ingest multiple documents with fewer network round trips, reducing overhead and latency.

This approach relies on newline-delimited JSON format, making careful formatting and error handling essential for stable indexing operations.

By batching large datasets efficiently, bulk indexing ensures that documents are rapidly ingested, allowing timely data availability for search and analytics.

1. Prepare a bulk request file (e.g., bulk_data.json) containing action lines followed by source lines.

   Each action line specifies an index operation and the subsequent line contains the document to be indexed.

2. Use the bulk endpoint with a POST request to send the bulk request.

   $ curl --request POST --header "Content-Type: application/json" --data-binary "@bulk_data.json" http://localhost:9200/_bulk
   {"took":10,"errors":false,"items":[{"index":{"_index":"my_index","_id":"1","result":"created","status":201}},...]}

   Check the response for errors; a value of false indicates all documents were processed successfully.

3. Refresh the index if immediate searchability is required.

   $ curl --request POST http://localhost:9200/my_index/_refresh
   {"_shards":{"total":1,"successful":1,"failed":0}}

   Large bulk requests may impact cluster performance. Consider splitting documents into manageable batches.

4. Monitor cluster health and indexing rate to ensure stability and optimize performance.

   Balancing request sizes and indexing speed prevents excessive load on the cluster.

5. Verify document counts and query results to confirm that all intended documents were successfully indexed.