Skip to content

4. Better retrieval

Intermediate · 14 min read

If the right chunk never reaches the prompt, the best LLM in the world can't answer. Retrieval is where most RAG quality is won or lost. The techniques on this page work on both sides of the search: before it (improve the query) and after it (improve the ranking).

flowchart LR
    Q[User question] --> RW[Rewrite / expand<br/>4.2 · 4.3 · 4.4]
    RW --> S[Hybrid search<br/>top 20–50]
    S --> RR[Rerank<br/>4.5]
    RR --> D[Diversify<br/>4.6]
    D --> P[Top 3–8 into the prompt]

4.1 Why first-pass retrieval misses

Cause Example Technique
Follow-up question with no subject "And for laptops?" Query rewriting (4.2)
User words ≠ document words "money back" vs "refund" Multi-query (4.3), HyDE (4.4), hybrid search
Question vs answer have different shapes Short question, long declarative chunk HyDE (4.4)
Right chunk retrieved but ranked 9th Similar-looking chunks crowd it out Reranking (4.5)
Top results are near-duplicates Three copies of the same policy MMR (4.6)
Exact codes and names missed "error E-4012", "Laptop Pro X2" Hybrid BM25 + vectors
Wrong product, year or tenant Retrieves the 2023 policy Metadata filters (4.7)

The examples below use the same small corpus and a bag-of-words stand-in for embeddings:

import math
import re
from collections import Counter

CHUNKS = {
    "c1": "Damaged items get a full refund including shipping.",
    "c2": "Refunds are processed within 5 working days of approval.",
    "c3": "Returns must be started within 30 days of delivery.",
    "c4": "Express shipping takes 1 day and costs 9 dollars.",
    "c5": "All laptops have a 1 year warranty covering hardware faults.",
    "c6": "Accidental damage such as a cracked screen is not covered by the warranty.",
    "c7": "To return an item, print the return label from your orders page.",
}
STOP = {"a", "an", "the", "is", "are", "to", "of", "and", "or", "in", "do", "does", "how", "what", "can", "i", "my",
        "it", "if", "for", "get", "be", "by", "from", "your", "yes", "you", "within", "all", "have", "such", "as", "back"}

def embed(text: str) -> Counter:
    words = re.findall(r"[a-z0-9]+", text.lower())
    return Counter(w.rstrip("s") if len(w) > 3 else w for w in words if w not in STOP)

def cosine(a: Counter, b: Counter) -> float:
    dot = sum(a[t] * b[t] for t in a)
    norm = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
    return dot / norm if norm else 0.0

VECTORS = {cid: embed(text) for cid, text in CHUNKS.items()}

def search(query: str, k: int = 3) -> list[tuple[str, float]]:
    q = embed(query)
    ranked = sorted(((cid, cosine(q, v)) for cid, v in VECTORS.items()), key=lambda x: -x[1])
    return [(cid, round(s, 2)) for cid, s in ranked[:k]]

print(search("My screen cracked, can I send it back for my money?"))
Output
[('c6', 0.38), ('c1', 0.0), ('c2', 0.0)]

Only c6 (the cracked screen isn't covered by the warranty) matches. The chunks the user also needs, c1 (damaged items get a full refund) and c7 (how to return an item), score zero, because the user said "money" and "send back", not "refund" and "return". The techniques below fix exactly this.

4.2 Query rewriting: make follow-ups standalone

In a chat, the user's latest message is often not a searchable query. "What about laptops?" only makes sense with the history. Before retrieving, ask a small, fast LLM to rewrite the latest message into a standalone search query:

# no-run — one cheap LLM call before retrieval
REWRITE_PROMPT = """Rewrite the user's last message as a standalone search query for a document search engine.
Use the conversation for context. Keep product names, codes and numbers exactly. Output only the query.

Conversation:
{history}

Last message: {question}"""

history = "User: How long is the warranty on phones?\nAssistant: Phones have a 2 year warranty [warranty.md#3]."
query = llm(REWRITE_PROMPT.format(history=history, question="What about laptops?"), model="small-fast-model")
# -> "How long is the warranty on laptops?"

Rules that keep rewriting safe:

  • Search with the rewritten query, but show the LLM the original question and history when answering.
  • Use a small model with temperature=0. It adds one round trip, often 200–500 ms.
  • Skip the rewrite for a conversation's first message, if it's already a full question.
  • Log both queries. A bad rewrite is a common hidden cause of bad answers.

4.3 Multi-query retrieval with RRF

One phrasing of a question finds some relevant chunks and misses others. Multi-query asks the LLM for three to five different phrasings, searches with each, and merges the result lists with reciprocal rank fusion (RRF), the same fusion used for hybrid search. A chunk that ranks well for several phrasings rises to the top.

question = "My screen cracked, can I send it back for my money?"
variants = [   # what an LLM returns when asked for 3 alternative search queries
    "Is accidental damage like a cracked screen covered by the warranty?",
    "Can I get a refund for a damaged item?",
    "How do I return an item?",
]

def rrf(result_lists: list[list[tuple[str, float]]], k: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for results in result_lists:
        for rank, (cid, _) in enumerate(results, start=1):
            scores[cid] = scores.get(cid, 0) + 1 / (k + rank)
    return sorted(((cid, round(s, 4)) for cid, s in scores.items()), key=lambda x: -x[1])

single = search(question)
fused = rrf([search(q) for q in [question, *variants]])
print("single query:", [cid for cid, _ in single])
print("multi-query: ", [cid for cid, _ in fused[:4]])
for cid, score in fused[:4]:
    print(f"   {cid} {score}  {CHUNKS[cid]}")
Output
single query: ['c6', 'c1', 'c2']
multi-query:  ['c1', 'c6', 'c7', 'c2']
   c1 0.0645  Damaged items get a full refund including shipping.
   c6 0.0328  Accidental damage such as a cracked screen is not covered by the warranty.
   c7 0.0323  To return an item, print the return label from your orders page.
   c2 0.032  Refunds are processed within 5 working days of approval.

The damaged-item refund c1 now ranks first, because two phrasings found it. The return instructions c7 appear for the first time. The single query found neither.

The cost is one extra LLM call plus several searches, which can run in parallel. Multi-query helps most with vague or multi-part questions. For a precise question ("What is error E-4012?"), it adds noise and latency.

4.4 HyDE: search with a hypothetical answer

A question and the chunk that answers it look different. The question is short and asks; the chunk is a longer statement. HyDE (Hypothetical Document Embeddings) asks the LLM to write a fake answer first, then searches with the embedding of that answer. The fake answer may be wrong on details, but it looks like the real answer chunk, using the same kind of vocabulary, so it lands near it in vector space.

question = "Can I get my money back if it arrived broken?"
hypothetical = ("Yes. If an item arrives damaged or broken you get a full refund, "
                "usually including the shipping cost.")          # written by an LLM; details may be wrong

print("question    :", search(question))
print("HyDE answer :", search(hypothetical))
Output
question    : [('c1', 0.0), ('c2', 0.0), ('c3', 0.0)]
HyDE answer : [('c1', 0.77), ('c4', 0.22), ('c2', 0.13)]

The question shares no words with any chunk, so every score is 0 and the "top 3" is arbitrary. The hypothetical answer uses the documents' own vocabulary ("damaged", "full refund", "shipping") and finds c1 with a clear lead. Real embedding models narrow this gap, but the effect is the same.

The fake answer is only a search key. Never show it to the user or put it in the final prompt. Trade-offs:

  • It adds an LLM call (latency and cost) before every search.
  • It helps most when users' wording differs from the documents' wording, and in zero-shot domains without a tuned embedding model.
  • It can hurt when the LLM knows nothing about the topic and invents a misleading answer, for example with internal product names. Keep the original query in the mix: search with both and fuse with RRF.

4.5 Reranking with a cross-encoder

Embedding search uses a bi-encoder. The question and every chunk are embedded separately, so search over millions of chunks is fast, but the model never looks at the question and chunk together. A cross-encoder reranker reads the question and a chunk together and outputs a relevance score. It's far more accurate, and far too slow to run over the whole corpus. So use both, in two stages:

flowchart LR
    Q[Question] --> B[Stage 1: bi-encoder / hybrid search<br/>millions of chunks → top 50<br/>fast, high recall]
    B --> C[Stage 2: cross-encoder reranker<br/>50 → top 5<br/>slow, high precision]
    C --> L[LLM prompt]
Bi-encoder (embeddings) Cross-encoder (reranker)
Input Question or chunk Question and chunk together
Precompute chunk vectors Yes, at indexing time No, every pair is scored at query time
Speed Milliseconds over millions (ANN index) ~10–100 ms per batch of pairs
Accuracy Good Better: sees word-level interactions between question and chunk
Role Recall: get the answer into the top 50 Precision: get it into the top 5 
# no-run — pip install sentence-transformers (open-source rerankers run on CPU for small batches)
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")          # or a hosted API: Cohere Rerank, Voyage, Jina

def retrieve_and_rerank(question: str, first_k: int = 50, final_k: int = 5) -> list[dict]:
    candidates = hybrid_search(question, k=first_k)         # stage 1: wide net
    scores = reranker.predict([(question, c["text"]) for c in candidates])
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [{**c, "rerank_score": float(s)} for c, s in ranked[:final_k]]

Reranking is usually the single biggest quality gain you can add to a basic pipeline. Tips:

  • Rerank 20–100 candidates. Too few and the right chunk isn't there to promote. Too many and latency grows.
  • Reranker scores are comparable across questions, so a threshold ("drop chunks below 0.2") is a good way to send fewer chunks or answer "I don't know".
  • An LLM can rerank too ("rank these 20 passages by relevance"). It's slower and costlier than a cross-encoder, but needs no extra model.

4.6 MMR: avoid five copies of the same chunk

Corpora are full of near-duplicates: the same policy in three versions, a paragraph copied into several pages. Plain top-k returns all of them, wasting prompt space. Maximal marginal relevance (MMR) picks results one at a time. Each pick balances relevance to the query against similarity to the chunks already chosen:

DUP_CHUNKS = {
    "a": "Refunds are processed within 5 working days of approval.",
    "b": "Refunds are processed within 5 working days after approval.",           # near-duplicate (older version)
    "c": "Refund processing takes 5 working days once approved.",                 # near-duplicate (FAQ copy)
    "d": "Refunds go back to the original payment method.",
    "e": "Refunds older than 30 days need a manager's approval.",
}
vecs = {cid: embed(t) for cid, t in DUP_CHUNKS.items()}

def mmr(query: str, k: int = 3, lam: float = 0.5) -> list[str]:
    q = embed(query)
    chosen, candidates = [], set(vecs)
    while candidates and len(chosen) < k:
        def score(cid):
            redundancy = max((cosine(vecs[cid], vecs[s]) for s in chosen), default=0)
            return lam * cosine(q, vecs[cid]) - (1 - lam) * redundancy
        best = max(sorted(candidates), key=score)
        chosen.append(best)
        candidates.remove(best)
    return chosen

query = "How are refunds processed?"
plain = sorted(vecs, key=lambda c: -cosine(embed(query), vecs[c]))[:3]
print("plain top-3:", plain, [DUP_CHUNKS[c][:45] for c in plain])
print("MMR top-3:  ", mmr(query), [DUP_CHUNKS[c][:45] for c in mmr(query)])
Output
plain top-3: ['a', 'b', 'd'] ['Refunds are processed within 5 working days o', 'Refunds are processed within 5 working days a', 'Refunds go back to the original payment metho']
MMR top-3:   ['a', 'd', 'e'] ['Refunds are processed within 5 working days o', 'Refunds go back to the original payment metho', "Refunds older than 30 days need a manager's a"]

Plain top-3 spends two of its three slots on the same fact (a and its older copy b). MMR keeps a, skips the copy, and uses the space for two different facts. lam=1 is plain relevance ranking and lam=0 is pure diversity. Values around 0.5–0.7 work well. Deduplicating at indexing time (content hashes, near-duplicate detection) is even better, because then the copies never get stored.

4.7 Metadata filters and query routing

Some questions need a filter, not a better similarity score. "What was the refund policy in 2024?" should only search 2024 documents. Two common patterns:

  • Self-query: an LLM extracts filters from the question with structured output, for example {"query": "refund policy", "year": 2024, "doc_type": "policy"}. Your code validates them against the allowed fields before searching. Never pass LLM output straight into a filter, and never let it set the tenant: that comes from the logged-in session.
  • Routing: classify the question and send it to the right source. Policies go to the vector index, order status goes to an API, revenue numbers go to text-to-SQL. Agentic RAG goes further and lets an agent decide what to search, and whether to search again.

4.8 Which technique, when?

Technique Fixes Added latency Added cost Add it when
Hybrid BM25 + vectors Exact terms, codes, names Low Low Almost always
Query rewriting Follow-up questions in chat 1 small LLM call Low Any multi-turn chat
Reranking Right chunk ranked too low 50–300 ms Low to medium Almost always, once you have an eval set
Multi-query + RRF Vague or multi-part questions 1 LLM call + parallel searches Medium Recall is low on complex questions
HyDE Vocabulary and shape mismatch 1 LLM call Medium Zero-shot domains, short questions
MMR / dedup Near-duplicate results Negligible None Corpora with versions or copies
Self-query filters Dates, products, doc types 1 small LLM call Low Questions carry constraints

Add them one at a time and keep each only if your eval set improves. Every technique adds latency, and stacking all of them blindly can make answers slower without making them better.

Interview questions

What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder embeds the query and documents separately into vectors, so document vectors are precomputed and nearest-neighbour search scales to millions. A cross-encoder takes the query and a document together and outputs a relevance score. It's more accurate because it models their interactions, but it must run per pair at query time. Hence two-stage retrieval: the bi-encoder (plus BM25) retrieves the top 50, and the cross-encoder reranks them to the top 5.

Explain HyDE. When does it help and when does it hurt?

Hypothetical Document Embeddings: an LLM writes a plausible answer to the question, and you search with that answer's embedding, because an answer looks more like the target chunk than a question does. It helps with vocabulary or shape mismatch and zero-shot domains. It costs an LLM call per query, and it can mislead when the LLM knows nothing about the domain. The hypothetical answer is only a search key and is never shown to the user.

How do you handle follow-up questions in a RAG chatbot?

Rewrite the latest message into a standalone query using the conversation history, with a small LLM call. Retrieve with the rewritten query, but generate with the full history and the original question. Log both, because bad rewrites are a common hidden failure.

Your retriever returns the right chunk at rank 12, but you only send 5. What do you do?

Add a reranker: retrieve 30–50 candidates with hybrid search, rerank with a cross-encoder, and send the top 5. Also check chunking (is the chunk diluted?), contextual headers, and hybrid search for exact terms. Measure recall@5 before and after on the eval set.

Practice

  • Add a fourth variant to the multi-query example and see how the fused ranking changes.
  • Run HyDE with a deliberately wrong hypothetical answer ("Express shipping is free.") and watch retrieval go wrong.
  • Change lam in mmr to 1.0 and 0.2 and compare the picks.

Next: 5. Parent–child retrieval & GraphRAG Back to: RAG overview · Notes overview