How to configure a text splitter in LlamaIndex

Retrieval quality in LlamaIndex depends on the nodes created before an index or retriever sees a document. A text splitter controls the token budget for each node and the context repeated at neighboring boundaries, so its settings affect both retrieval precision and the amount of source text available for an answer.

The SentenceSplitter class favors complete sentence and paragraph boundaries while enforcing token-based chunk_size and chunk_overlap values. Assigning it to Settings.text_splitter makes that instance the active default for components created later in the same Python process.

A credential-free validation can use llama-index-core with one in-memory Document and no model or vector store. Representative source text matters because short or uniform samples can hide boundary problems that appear in long tables, lists, or dense paragraphs.

1. Create sentence_splitter_check.py with the imports and representative source text.

   sentence_splitter_check.py

   from importlib.metadata import version
    
   from llama_index.core import Document, Settings
   from llama_index.core.node_parser import SentenceSplitter
   from llama_index.core.utils import get_tokenizer
    
    
   sample_text = (
       "Refund intake records the ticket number and requested credit. "
       "Billing review confirms the invoice status and approval owner. "
       "Manager approval records the customer message and follow-up date. "
       "Shipping exceptions retain the carrier notes and delivery estimate. "
       "Final archive keeps the refund decision and audit timestamp. "
       "Escalation notes list the support queue and backup approver. "
       "Resolution summaries capture the promised response date. "
       "Quality review compares the outcome with the original policy."
   )

2. Add the active SentenceSplitter configuration beneath sample_text.

   splitter = SentenceSplitter(chunk_size=64, chunk_overlap=16)
   Settings.text_splitter = splitter
   nodes = Settings.text_splitter.get_nodes_from_documents(
       [Document(text=sample_text)]
   )

   Both values are token counts, and chunk_overlap must remain smaller than chunk_size. Production values such as 512 and 64 reduce the number of small nodes while retaining some boundary context.

3. Append fail-capable node-count and token-limit checks below the splitter call.

   tokenizer = get_tokenizer()
   token_counts = [
       len(tokenizer(node.get_content(metadata_mode="none")))
       for node in nodes
   ]
    
   assert Settings.text_splitter is splitter
   assert len(nodes) > 1
   assert max(token_counts) <= splitter.chunk_size
    
   print(f"llama-index-core={version('llama-index-core')}")
   print(f"active_text_splitter={Settings.text_splitter.class_name()}")
   print(f"chunk_size={splitter.chunk_size}")
   print(f"chunk_overlap={splitter.chunk_overlap}")
   print(f"node_count={len(nodes)}")
   print("node_token_counts=" + ",".join(map(str, token_counts)))
   print("status=PASS")

4. Review the consolidated sentence_splitter_check.py before execution.

   sentence_splitter_check.py

   from importlib.metadata import version
    
   from llama_index.core import Document, Settings
   from llama_index.core.node_parser import SentenceSplitter
   from llama_index.core.utils import get_tokenizer
    
    
   sample_text = (
       "Refund intake records the ticket number and requested credit. "
       "Billing review confirms the invoice status and approval owner. "
       "Manager approval records the customer message and follow-up date. "
       "Shipping exceptions retain the carrier notes and delivery estimate. "
       "Final archive keeps the refund decision and audit timestamp. "
       "Escalation notes list the support queue and backup approver. "
       "Resolution summaries capture the promised response date. "
       "Quality review compares the outcome with the original policy."
   )
    
   splitter = SentenceSplitter(chunk_size=64, chunk_overlap=16)
   Settings.text_splitter = splitter
   nodes = Settings.text_splitter.get_nodes_from_documents(
       [Document(text=sample_text)]
   )
    
   tokenizer = get_tokenizer()
   token_counts = [
       len(tokenizer(node.get_content(metadata_mode="none")))
       for node in nodes
   ]
    
   assert Settings.text_splitter is splitter
   assert len(nodes) > 1
   assert max(token_counts) <= splitter.chunk_size
    
   print(f"llama-index-core={version('llama-index-core')}")
   print(f"active_text_splitter={Settings.text_splitter.class_name()}")
   print(f"chunk_size={splitter.chunk_size}")
   print(f"chunk_overlap={splitter.chunk_overlap}")
   print(f"node_count={len(nodes)}")
   print("node_token_counts=" + ",".join(map(str, token_counts)))
   print("status=PASS")

5. Run sentence_splitter_check.py in the active Python environment.

   $ python3 sentence_splitter_check.py
   llama-index-core=0.14.23
   active_text_splitter=SentenceSplitter
   chunk_size=64
   chunk_overlap=16
   node_count=2
   node_token_counts=52,41
   status=PASS

   The run fails before status=PASS when the active splitter is replaced, the sample stays in one node, or any emitted node exceeds the configured token budget. Re-index documents after changing splitter values because existing nodes and embeddings keep their earlier boundaries.