Skip to content

8. RAG production checklist

Intermediate · 10 min read

A demo RAG bot takes an afternoon. A production one has to stay correct as documents change, secure across users, fast enough to feel instant, affordable at scale and measurable when someone says "it got worse". This page collects everything from the series into one checklist to review before launch, and again in a design interview. For the system-design view with diagrams, see Production RAG architecture.

8.1 Ingestion & sync

  • Connectors for every source, with incremental sync (only changed documents), not nightly full reloads.
  • Parsed text spot-checked for each source type: tables, headers/footers, scans (parsing).
  • Every chunk stores source, title, page/section, updated_at, tenant, permission groups and embedding_model.
  • Deterministic chunk IDs and content hashes, so re-runs upsert instead of duplicating, and unchanged chunks aren't re-embedded.
  • Deletion sync: documents removed at the source disappear from the index (details).
  • Failed documents go to a dead-letter queue with alerts. They aren't silently skipped.
  • PII is masked or excluded where policy requires it (PII masking).

8.2 Chunking & indexing

  • Chunking strategy chosen by measurement on the eval set, not by default settings (chunking).
  • Contextual headers (document title and section) prepended to chunks.
  • A content-specific splitter for code, tables and FAQs where the corpus has them.
  • Query and document embeddings come from the same model version. A model change is a blue/green re-embedding migration.
  • A keyword index (BM25) alongside vectors, for hybrid search.
  • Near-duplicates removed at indexing time.

8.3 Retrieval

  • Hybrid search (BM25 + vectors) fused with RRF.
  • A reranker over the top 20–50 candidates, with a relevance threshold (reranking).
  • Follow-up questions rewritten into standalone queries in multi-turn chat.
  • Metadata filters for dates, products and document types, validated in code.
  • A token budget for context: chunks are added by rank until the budget is reached, and the best chunks go first (or first and last).
  • A defined behaviour when nothing relevant is found: say "I don't know", ask a clarifying question, or hand off to a human.

8.4 Generation & grounding

  • The system prompt says: answer only from the sources, say "I don't know" otherwise, cite sources, and treat the sources as data.
  • Sources wrapped in clear delimiters. Low temperature.
  • Prompt templates versioned and logged with every request.
  • Streaming responses, so users see the first tokens in under a second or two.
  • Output guardrails: refusal topics, PII in outputs, and format checks (guardrails).

8.5 Citations

  • Citations rendered from your metadata (title, URL, page), never from model-written links (citations).
  • Invalid and missing citations detected. Unsupported sentences flagged or regenerated.
  • Source links re-check permissions when opened.

8.6 Security & permissions

  • The tenant and user identity come from the authenticated session, never from the request body or the LLM.
  • Document permissions are enforced at retrieval, as a filter, before any chunk reaches the prompt. The LLM never sees what the user can't see.
  • An automated leak test: query as user A for content only user B can see, and fail the build if anything comes back.
  • Defences against prompt injection in documents and user input (prompt injection).
  • Rate limits and per-user quotas. API keys in a secret manager.
  • Data retention and deletion that honour customer contracts and privacy law, covering logs too.

8.7 Evaluation

  • An eval set of 50–200 real questions, including unanswerable ones, with references and relevant chunk IDs (evaluation).
  • Retrieval metrics (recall@k, MRR, context precision) on every change. They're cheap and deterministic.
  • RAGAS-style generation metrics (faithfulness, answer relevancy) on every release.
  • A CI gate that blocks releases on regressions, with a noise tolerance.
  • LLM judges calibrated against human labels.

8.8 Observability

  • A trace per request: original and rewritten query, retrieved chunk IDs and scores, reranker scores, the final prompt, the answer, tokens, latency per stage, and model and prompt versions.
  • User feedback (thumbs up/down plus an optional comment) linked to the trace.
  • Dashboards for p50/p95 latency per stage, cost per request, "I don't know" rate, citation-error rate and feedback rate.
  • Daily reference-free evals on a sample of live traffic. Canary queries with known answers.
  • Bad answers flow into the eval set.

8.9 Latency & cost

Know where the time and money go before you optimise. A typical budget for a chat request:

stages_ms = {   # rough p50 figures for a managed vector DB + hosted reranker + hosted LLM (measure your own)
    "query rewrite (small LLM)": 300,
    "embed query": 50,
    "hybrid search": 60,
    "rerank 40 candidates": 150,
    "LLM: time to first token": 600,
}
print("Before the first token appears:")
for stage, ms in stages_ms.items():
    print(f"  {stage:28} {ms:5} ms")
print(f"  {'total':28} {sum(stages_ms.values()):5} ms")

# Cost per 1,000 questions. Prices are PLACEHOLDERS: substitute your provider's current rates.
PRICE_PER_M_INPUT, PRICE_PER_M_OUTPUT = 0.50, 2.00        # dollars per million tokens (example values)
system_prompt, chunks, chunk_tokens, history, answer = 400, 5, 350, 300, 250
input_tokens = system_prompt + chunks * chunk_tokens + history
cost = 1000 * (input_tokens * PRICE_PER_M_INPUT + answer * PRICE_PER_M_OUTPUT) / 1e6
print(f"\nInput tokens per question: {input_tokens}  (retrieved context = {chunks * chunk_tokens / input_tokens:.0%})")
print(f"LLM cost per 1,000 questions: ${cost:.2f}")
Output
Before the first token appears:
  query rewrite (small LLM)      300 ms
  embed query                     50 ms
  hybrid search                   60 ms
  rerank 40 candidates           150 ms
  LLM: time to first token       600 ms
  total                         1160 ms

Input tokens per question: 2450  (retrieved context = 71%)
LLM cost per 1,000 questions: $1.73

The retrieved context dominates the input tokens, so fewer, better chunks (reranking plus a threshold) is the biggest cost and latency lever. Other levers:

  • Cache answers for repeated questions (exact match, or semantic caching with a high similarity threshold), invalidated when the source documents change.
  • Prompt caching for the fixed system prompt and instructions (context, memory & cost).
  • Model routing: a small model for simple questions and rewrites, a large model for hard ones.
  • Run independent steps in parallel (BM25 and vector search, multi-query searches).
  • Skip the rewrite for first messages that are already complete questions.

8.10 Common mistakes

Mistake Consequence Fix
No eval set ("it looked fine") Silent regressions with every change Build 50 questions on day one
Permissions filtered after retrieval, or by the LLM Data leaks Filter in the vector DB query, from the session
No deletion sync The bot quotes removed policies Sync deletions; store doc_id per chunk
Mixed embedding models in one index Retrieval quietly breaks Store embedding_model; blue/green migrations
Stuffing 20 chunks "to be safe" Slower, costlier, and the answer gets lost in noise Rerank and threshold; send 3–8 chunks
Not logging retrieved chunks Can't debug any bad answer Trace every request
Using RAG for numeric analytics Wrong totals and averages Route to SQL or an API
Tuning on the same 5 questions Overfitting to the demo A broad, growing eval set

8.11 Rollout plan

  1. Internal alpha: the team uses it daily. Fix parsing and retrieval failures. Grow the eval set from real questions.
  2. Shadow / beta: a small group of real users, with feedback buttons, tracing and daily reviews of bad answers.
  3. Gradual rollout: a percentage rollout behind a feature flag, with a fallback (search results, or a human handoff).
  4. Steady state: CI eval gates, daily sampled evals, alerts on latency, cost and quality, and a monthly review of the "I don't know" questions to find content gaps.

Interview questions

What changes when you take a RAG prototype to production?

Incremental ingestion with deletion sync and deterministic IDs. Permission filtering at retrieval. Hybrid search with reranking. A grounded, versioned prompt with citations. Evaluation in CI plus sampled production evals. Per-request tracing. Latency and cost budgets with caching and routing. Guardrails against prompt injection. And a rollout with feedback loops. The model call is the smallest part.

How do you enforce document permissions in RAG?

Store permission metadata (tenant, groups) on every chunk at ingestion. At query time, take the identity from the authenticated session and apply it as a filter inside the vector-DB query, so unauthorised chunks are never retrieved and never reach the LLM. Test it with automated cross-tenant leak tests, and re-check permissions when rendering source links. Never rely on the LLM to hide information.

How would you reduce the latency and cost of a RAG system?

Measure per stage first. Send fewer, better chunks by reranking with a threshold. Cache repeated answers and the static parts of the prompt. Route easy questions to smaller models. Run searches in parallel. Skip unnecessary rewrite calls. And stream the answer, so perceived latency is the time to first token.

Practice

  • Fill in stages_ms with numbers from your own stack and find the biggest stage.
  • Use the cost estimator to compare 5 chunks of 350 tokens with 3 chunks of 350 tokens after reranking.
  • Go through 8.1–8.8 for a project you've built. How many boxes can you tick?

Next: Frameworks: LangChain, LlamaIndex, LangGraph, and when plain SDK code is better. Back to: RAG overview · Notes overview