Most developers build their first Retrieval-Augmented Generation (RAG) system in an afternoon. You read a tutorial, load some documentation pages, run a standard sentence splitter (500 tokens, 10% overlap), upload them to a vector database, and call it a day.
This works perfectly on clean, simple test queries. But once real customers start typing fragmented search terms, using slang, or referencing tables buried deep inside PDFs, the naive RAG setup crumbles.
During the initial rollout of our document training pipeline, we watched the agent fail to answer simple questions because the matching chunk was missing crucial context from the preceding header.
Iteration 1: Naive Token Splitting (The Tutorial Way)
Our first setup divided files into standard token blocks. The problem? Key sentences were routinely sliced in half. A paragraph explaining refund criteria would have its constraints in Chunk A and its approval steps in Chunk B. The vector search would fetch Chunk B, but without Chunk A's context, the LLM hallucinated the criteria.
Iteration 2: Recursive Paragraph & Header Preservation
We moved to structural markdown parsing. We split documents by headers (#, ##, ###) and kept list items together. This was a massive improvement, but it introduced a new issue: document size skew. A huge section would create a single giant chunk, diluting the semantic vector representation and crowding the prompt window.
Iteration 3: The Winner - Parent-Child Node Mapping
The solution that finally brought our retrieval accuracy to 94.2% was decoupling the search representation from the generation context.
We chunked documents into small, highly specific "Child" nodes (100 tokens each) for vector indexing. However, each child node references a larger "Parent" node (1000 tokens) representing the surrounding context.
// Decoupled Retrieval Pattern
{
"child_node_id": "child-9418",
"vector_embedding": [...],
"search_text": "Refunds are processed within 3 business days.",
"parent_context_ref": "parent-doc-refund-policy-section"
}When a query hits the child node, we retrieve and feed the entire parent context block to the LLM. The search remains precise, but the generation model has full, uninterrupted context to formulate its final reply. Query latency dropped from 980ms to 240ms.