3. Chunking for RAG

Intermediate · 11 min read

You can't embed a 40-page PDF as one vector. Its meaning averages out to mush, and it wouldn't fit in the prompt anyway. So documents are split into chunks, and a chunk is the unit you retrieve, cite and send to the LLM. Chunking quietly decides whether the answer can be found at all.

The basics are in the NLP notes

NLP for GenAI already builds the three core splitters from scratch: fixed-size token chunks with overlap, recursive chunking and structure-aware chunks with metadata. This page is about choosing between them and the techniques that build on them.

3.1 The chunk-size trade-off

	Small chunks (≈100–200 tokens)	Large chunks (≈800–1500 tokens)
Embedding	Precise: one idea per vector	Diluted: several topics averaged together
Retrieval	Finds the exact sentence	Can miss a fact buried in a long chunk
Context for the LLM	May lack context ("It costs $9", but what does?)	Rich context
Prompt cost	Low per chunk, so you can send more of them	High, so fewer fit
Good for	FAQs, short facts, precise lookups	Explanations, procedures, legal clauses

A sensible default is around 300–500 tokens with 10–15% overlap, split recursively on paragraphs and sentences. Then measure on your own questions. Parent–child retrieval gets you both: match on small chunks, send big ones.

3.2 Why overlap matters

A fixed-size splitter cuts wherever the count runs out, often in the middle of the one sentence that answers the question. Overlap repeats the end of each chunk at the start of the next, so a fact near a boundary appears whole in at least one chunk:

text = ("Standard shipping takes 3 to 5 days. We ship to India, the UK and the US. "
        "Express shipping takes 1 day and costs 9 dollars. Orders over 50 dollars ship free.")
answer = "Express shipping takes 1 day and costs 9 dollars."

def word_chunks(text: str, size: int, overlap: int) -> list[str]:
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

for overlap in (0, 8):
    chunks = word_chunks(text, size=12, overlap=overlap)
    intact = any(answer in c for c in chunks)
    print(f"overlap={overlap}: {len(chunks)} chunks, answer intact in one chunk: {intact}")
    for c in chunks:
        print("   ", c)

Output

overlap=0: 3 chunks, answer intact in one chunk: False
    Standard shipping takes 3 to 5 days. We ship to India, the
    UK and the US. Express shipping takes 1 day and costs 9
    dollars. Orders over 50 dollars ship free.
overlap=8: 6 chunks, answer intact in one chunk: True
    Standard shipping takes 3 to 5 days. We ship to India, the
    to 5 days. We ship to India, the UK and the US.
    ship to India, the UK and the US. Express shipping takes 1
    UK and the US. Express shipping takes 1 day and costs 9
    Express shipping takes 1 day and costs 9 dollars. Orders over 50
    day and costs 9 dollars. Orders over 50 dollars ship free.

Without overlap, the answer sentence is cut in two. Neither half matches "how much is express shipping" well, and neither half alone gives the LLM the full fact. With 8 words of overlap, chunk 5 holds the whole sentence. The price is twice as many chunks to embed and store. The overlap here is extreme to make the effect visible. 10–15% is normal. Splitting on sentence boundaries (recursive chunking) prevents most of these cuts. Overlap is the safety net for the rest.

3.3 Semantic chunking

Recursive splitting follows the formatting (paragraphs, sentences). Semantic chunking follows the meaning. It embeds each sentence and starts a new chunk where neighbouring sentences stop being similar, which marks a topic change. It helps most with text that has weak formatting, like transcripts, emails or OCR output.

import math
import re
from collections import Counter

STOP = {"a", "an", "the", "is", "are", "to", "of", "and", "in", "it", "for", "on", "by", "be", "can", "you", "your",
        "within", "after", "all", "we", "our"}

def embed(text: str) -> Counter:          # stand-in for a sentence-embedding model
    words = re.findall(r"[a-z]+", text.lower())
    return Counter(w.rstrip("s") for w in words if w not in STOP)

def cosine(a: Counter, b: Counter) -> float:
    dot = sum(a[t] * b[t] for t in a)
    norm = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
    return dot / norm if norm else 0.0

transcript = ("Your refund is processed after approval. Refund approval takes two days. "
              "A refund goes back to the original card. Now about shipping. Express shipping costs nine dollars. "
              "Standard shipping is free on large orders. Finally the warranty. The warranty lasts one year.")

def semantic_chunks(text: str, threshold: float = 0.1) -> list[str]:
    sentences = re.split(r"(?<=\.)\s+", text.strip())
    vectors = [embed(s) for s in sentences]
    chunks, current = [], [sentences[0]]
    for prev, vec, sent in zip(vectors, vectors[1:], sentences[1:]):
        similarity = cosine(prev, vec)
        if similarity < threshold:                 # topic shift: close the chunk
            chunks.append(" ".join(current))
            current = []
        current.append(sent)
    chunks.append(" ".join(current))
    return chunks

for i, c in enumerate(semantic_chunks(transcript), start=1):
    print(f"chunk {i}: {c}")

Output

chunk 1: Your refund is processed after approval. Refund approval takes two days. A refund goes back to the original card.
chunk 2: Now about shipping. Express shipping costs nine dollars. Standard shipping is free on large orders.
chunk 3: Finally the warranty. The warranty lasts one year.

Real implementations compare each sentence with a window of neighbours, and set the threshold from the distribution of similarities (for example, split at the lowest 10%) instead of a fixed number. Semantic chunking costs an embedding call per sentence at indexing time. In published comparisons it often does only a little better than good recursive chunking. Try it if your documents lack structure, and keep it only if your evals improve.

3.4 Contextual chunk headers

A chunk cut out of a document loses its surroundings. "It costs 9 dollars and takes 1 day" doesn't say what costs 9 dollars, so neither the embedding model nor the LLM can tell. The cheap fix is to prepend the document title and section path to each chunk before embedding:

sections = [  # output of a structure-aware splitter: (heading path, chunk text)
    ("Shipping FAQ > Express shipping", "It costs 9 dollars and takes 1 day."),
    ("Shipping FAQ > Standard shipping", "It is free on orders over 50 dollars."),
    ("Warranty > Laptops", "It lasts one year and covers hardware faults."),
]
query = embed("How much does express shipping cost?")

for use_header in (False, True):
    texts = [f"{path}\n{body}" if use_header else body for path, body in sections]
    scores = [round(cosine(query, embed(t)), 2) for t in texts]
    print(f"headers={use_header!s:5}  scores={scores}")

Output

headers=False  scores=[0.2, 0.0, 0.0]
headers=True   scores=[0.52, 0.26, 0.0]

Without headers, the right chunk has almost no words in common with the question. With headers, it ranks first. Store the header in the embedded text, and in metadata too, so you can show the section in citations.

Contextual retrieval takes this further. An LLM writes one or two sentences per chunk explaining where it sits in the whole document ("This chunk is from Acme's 2026 shipping FAQ and describes express delivery pricing."), and that text is prepended before embedding and BM25 indexing. Anthropic reported it cut retrieval failures by about a third, and by more with reranking. It costs one LLM call per chunk at indexing time. Prompt caching the full document makes that affordable.

# no-run — contextual retrieval: one cheap LLM call per chunk, at indexing time
PROMPT = """<document>
{document}
</document>
Here is a chunk from the document above:
<chunk>
{chunk}
</chunk>
Write 1-2 sentences that situate this chunk within the overall document, to improve search retrieval.
Answer with only those sentences."""

context = llm(PROMPT.format(document=full_text, chunk=chunk_text))     # cache the document part of the prompt
embedded_text = f"{context}\n\n{chunk_text}"                           # embed and BM25-index this

3.5 Content that needs its own splitter

Content	Split by	Watch out for
Markdown / HTML docs	Headings, then paragraphs	Keep the heading path as a header
PDFs	Page, then paragraphs	Keep page numbers. Repair broken lines first (parsing)
Source code	Functions and classes (AST or tree-sitter), not characters	Include the file path and class name
Tables	Whole small tables, or one row per chunk with column names	Never split a row in half
FAQs	One question + answer per chunk	Embedding the question text helps matching
Chat logs, transcripts	Conversation turns or time windows, or semantic chunking	Keep speaker names and timestamps
Legal / contracts	Clauses and sub-clauses	Cross-references ("as defined in 4.2") need the referenced clause

3.6 How to choose, in practice

1. Start with structure-aware or recursive splitting at ~400 tokens, ~50 overlap, with contextual headers.
2. Build a small eval set of real questions with their answer chunks (evaluation).
3. Try two or three variants (smaller/larger, semantic, parent–child) and compare recall@k and answer quality.
4. Keep chunk IDs deterministic, so re-chunking is a clean migration.

Interview questions

How do you choose chunk size and overlap?

It's a trade-off: small chunks embed precisely but lack context, large chunks have context but dilute the embedding and cost more prompt tokens. Start around 300–500 tokens with 10–15% overlap, split on document structure or sentences, and tune on an eval set measuring recall@k and answer quality. Different content types need different splitters. Parent–child retrieval avoids choosing one size for both matching and context.

What is semantic chunking, and when is it worth it?

Embed consecutive sentences and start a new chunk where their similarity drops, which marks a topic change. It helps on poorly structured text such as transcripts or OCR output. It costs extra embedding calls at indexing time, and on well-structured documents it often matches recursive chunking, so adopt it only if evals improve.

A chunk says 'It costs $9' and retrieval misses it. What do you do?

The chunk lost its context. Prepend the document title and section heading to each chunk before embedding (contextual chunk headers), or generate a short LLM-written context per chunk (contextual retrieval). Also add hybrid BM25 search so exact terms match, and consider parent–child retrieval so the LLM sees the surrounding text.

Practice

• In 3.2, find the smallest overlap that keeps the answer intact.
• Lower and raise the threshold in semantic_chunks and watch the chunks merge and split.
• Take one of your own Markdown documents and produce (heading path, chunk) pairs like in 3.4.

Next: 4. Better retrieval Back to: RAG overview · Notes overview