Skip to content

7. Evaluation with RAGAS

Intermediate · 15 min read

"It looks good on the five questions I tried" is how most RAG projects ship, and how most of them regress. Every page in this series ends with "keep it if your evals improve". This page builds those evals. The key idea is to measure retrieval and generation separately, so a bad score tells you which stage to fix.

7.1 What to measure

Every RAG evaluation sample has up to four parts:

Field (RAGAS name) What it is Where it comes from
user_input The question Eval set
retrieved_contexts The chunks your retriever returned Your pipeline, at eval time
response Your system's answer Your pipeline, at eval time
reference The correct answer (ground truth) Eval set, written or reviewed by a human

Each metric compares two or three of them:

flowchart LR
    Q[Question] -- answer relevancy --> A[Response]
    C[Retrieved contexts] -- faithfulness --> A
    Q -- context precision --> C
    R[Reference answer] -- context recall --> C
    R -- factual correctness --> A
Metric Question it answers Stage Needs a reference?
Context precision Are the relevant chunks ranked at the top? Retrieval Yes, or uses the response
Context recall Did retrieval find everything needed to answer? Retrieval Yes
Faithfulness Is every claim in the answer supported by the contexts? (no hallucination) Generation No
Answer relevancy Does the answer address the question that was asked? Generation No
Factual correctness Does the answer match the reference? End to end Yes

Faithfulness and answer relevancy need no reference answer, so you can run them on live production traffic.

7.2 Build the eval set

The eval set matters more than the tool. Aim for 50–200 questions to start:

  • Real questions from logs, support tickets and users. They're the most valuable kind.
  • Cover the types: simple lookups, multi-part, multi-hop, comparisons, follow-ups, and unanswerable questions where the right answer is "I don't know".
  • Label the relevant chunk IDs for each question, as well as the reference answer. That makes retrieval metrics cheap and exact.
  • Synthetic questions from an LLM can fill gaps. Give it a chunk and ask for a question it answers. RAGAS has a test-set generator for this. Have a human review a sample, because synthetic questions tend to be too easy and reuse the chunk's exact wording.
EVAL_SET = [
    {"question": "How long do refunds take?", "relevant": {"refunds#0"},
     "reference": "Refunds are processed within 5 working days of approval."},
    {"question": "Is a cracked screen covered by the warranty?", "relevant": {"warranty#1"},
     "reference": "No. Accidental damage such as a cracked screen is not covered."},
    {"question": "What does express shipping cost and how fast is it?", "relevant": {"shipping#1", "shipping#2"},
     "reference": "Express shipping costs 9 dollars and takes 1 day."},
]
print(len(EVAL_SET), "questions,", sum(len(s["relevant"]) for s in EVAL_SET), "labelled relevant chunks")
Output
3 questions, 4 labelled relevant chunks

See Evaluation & guardrails for versioning and growing the set over time.

7.3 Retrieval metrics you can compute yourself

With labelled chunk IDs, retrieval metrics need no LLM at all. They're instant, free and deterministic, so run them on every change. NLP for GenAI covers recall@k and MRR. Here they compare two versions of a retriever, along with context precision as RAGAS defines it: the average precision at each rank where a relevant chunk appears.

RUNS = {   # retrieved chunk IDs per question, in rank order, for two versions of the retriever
    "v1 vectors only": [["refunds#2", "refunds#0", "shipping#0"],
                        ["warranty#0", "returns#1", "warranty#3"],
                        ["shipping#1", "shipping#0", "returns#0"]],
    "v2 hybrid + rerank": [["refunds#0", "refunds#2", "refunds#1"],
                           ["warranty#1", "warranty#0", "returns#1"],
                           ["shipping#1", "shipping#2", "shipping#0"]],
}

def recall_at_k(retrieved: list[str], relevant: set[str]) -> float:
    return len(relevant & set(retrieved)) / len(relevant)

def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    return next((1 / r for r, cid in enumerate(retrieved, start=1) if cid in relevant), 0.0)

def context_precision(retrieved: list[str], relevant: set[str]) -> float:
    hits, total = 0, 0.0
    for rank, cid in enumerate(retrieved, start=1):
        if cid in relevant:
            hits += 1
            total += hits / rank             # precision@rank, counted only at relevant positions
    return total / hits if hits else 0.0

mean = lambda xs: sum(xs) / len(xs)
print(f"{'retriever':20} {'recall@3':>9} {'MRR':>6} {'ctx precision':>14}")
for name, runs in RUNS.items():
    rel = [s["relevant"] for s in EVAL_SET]
    print(f"{name:20} {mean([recall_at_k(r, g) for r, g in zip(runs, rel)]):9.2f}"
          f" {mean([reciprocal_rank(r, g) for r, g in zip(runs, rel)]):6.2f}"
          f" {mean([context_precision(r, g) for r, g in zip(runs, rel)]):14.2f}")
Output
retriever             recall@3    MRR  ctx precision
v1 vectors only           0.50   0.50           0.50
v2 hybrid + rerank        1.00   1.00           1.00

v1 misses the cracked-screen chunk entirely, half-finds the shipping answer, and ranks the refund answer second. v2 finds everything and ranks it first. Numbers like these settle "did reranking help?" in seconds, and they need no LLM.

7.4 The RAGAS metrics

When chunks aren't labelled, or to judge the answer itself, RAGAS uses an LLM as a judge in carefully broken-down steps. Knowing how each metric is computed tells you what a low score means.

Faithfulness: the share of the answer's claims that the retrieved contexts support.

  1. An LLM splits the answer into atomic claims.
  2. For each claim, an LLM judges whether the contexts support it (an NLI-style yes/no).
  3. Score = supported claims ÷ total claims.
contexts = ["Refunds are processed within 5 working days of approval.",
            "Damaged items get a full refund including shipping."]
response = "Refunds take 5 working days after approval. Damaged items are refunded in full, and you get a voucher."

claims = [   # step 1 and step 2 are LLM calls; here their outputs are written out
    ("Refunds take 5 working days after approval.", True),
    ("Damaged items are refunded in full.", True),
    ("Customers with damaged items also get a voucher.", False),    # not in any context: a hallucination
]
faithfulness = sum(ok for _, ok in claims) / len(claims)
print(f"faithfulness = {sum(ok for _, ok in claims)}/{len(claims)} = {faithfulness:.2f}")
for claim, ok in claims:
    print(f"  {'✓' if ok else '✗'} {claim}")
Output
faithfulness = 2/3 = 0.67
  ✓ Refunds take 5 working days after approval.
  ✓ Damaged items are refunded in full.
  ✗ Customers with damaged items also get a voucher.

Answer relevancy (called ResponseRelevancy in recent versions): an LLM writes a few questions that the answer would be a good answer to. The score is their average embedding similarity to the real question. An answer that drifts off topic, or dodges the question, produces generated questions that don't match. Answer relevancy does not check correctness. A confident wrong answer to the right question scores high, which is why it's paired with faithfulness.

Context precision: the rank-weighted measure from 7.3. RAGAS can judge each chunk's relevance with an LLM against the reference, or against the response when there's no reference.

Context recall: the reference answer is split into claims, and each is checked against the retrieved contexts. The score is the share of reference claims the contexts can support. Low recall means the answer couldn't be complete, whatever the LLM did.

7.5 Running RAGAS

# no-run — pip install ragas langchain-openai   (needs an API key for the judge LLM)
# Written for the ragas 0.2/0.3 API. Metric and class names change between releases: check docs.ragas.io.
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas import EvaluationDataset, evaluate
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import Faithfulness, LLMContextPrecisionWithReference, LLMContextRecall, ResponseRelevancy

samples = []
for item in EVAL_SET:
    result = rag_pipeline(item["question"])                   # your system: returns answer + retrieved chunks
    samples.append({
        "user_input": item["question"],
        "retrieved_contexts": [c["text"] for c in result["chunks"]],
        "response": result["answer"],
        "reference": item["reference"],
    })

judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-small"))

scores = evaluate(
    dataset=EvaluationDataset.from_list(samples),
    metrics=[LLMContextPrecisionWithReference(), LLMContextRecall(), Faithfulness(), ResponseRelevancy()],
    llm=judge,
    embeddings=embeddings,
)
print(scores)                     # averages, e.g. {'faithfulness': 0.91, 'context_recall': 0.84, ...}
df = scores.to_pandas()           # one row per question: read the worst rows, not just the averages

RAGAS isn't the only option. DeepEval, TruLens, Arize Phoenix, LangSmith and promptfoo cover similar metrics. The concepts on this page transfer, whatever the tool.

7.6 From scores to fixes

Averages tell you whether something is wrong. Sort by the worst-scoring questions and read them to find out what.

Low metric Meaning Look at Typical fixes
Context recall The needed information never reached the prompt Is it in the corpus? Is it parsed? Which rank does it get? Parsing, chunking, hybrid search, higher k, query rewriting
Context precision The right chunks are there, but buried under noise The rank of the relevant chunks Reranking, fewer chunks, metadata filters
Faithfulness The model adds claims that aren't in the sources The unsupported claims Stricter grounding prompt, citations, lower temperature, stronger model
Answer relevancy The answer dodges or drifts The response vs the question Prompt clarity, query rewriting, too much irrelevant context
Recall high, correctness low Good context, wrong answer The response vs the reference Generation: the prompt, chunk order, the model

7.7 Evals in CI and production

Treat the eval set like a test suite. Run it on every change to prompts, chunking, models or retrieval, and block the release if quality drops:

BASELINE = {"context_recall": 0.86, "context_precision": 0.81, "faithfulness": 0.93, "answer_relevancy": 0.88}
CANDIDATE = {"context_recall": 0.91, "context_precision": 0.84, "faithfulness": 0.87, "answer_relevancy": 0.89}
MIN = {"faithfulness": 0.90}      # hard floors, whatever the baseline
MAX_DROP = 0.03                   # tolerated noise: LLM-judged scores wobble between runs

failures = []
for metric, base in BASELINE.items():
    new = CANDIDATE[metric]
    flag = ""
    if new < base - MAX_DROP:
        flag = "REGRESSION"
    if new < MIN.get(metric, 0):
        flag = "BELOW FLOOR"
    if flag:
        failures.append(metric)
    print(f"{metric:18} {base:.2f} → {new:.2f}  {flag}".rstrip())
print("RESULT:", "FAIL — block the release" if failures else "PASS")
Output
context_recall     0.86 → 0.91
context_precision  0.81 → 0.84
faithfulness       0.93 → 0.87  BELOW FLOOR
answer_relevancy   0.88 → 0.89
RESULT: FAIL — block the release

The new version retrieves better, but its answers became less faithful. That's exactly the trade-off a single "overall quality" number would hide.

In production:

  • Run reference-free metrics (faithfulness, answer relevancy) on a sample of live traffic every day.
  • Track user feedback (thumbs up/down), "I don't know" rates and citation errors next to them.
  • Add every bad answer from production to the eval set, with a reference. The set grows from real failures.

7.8 Pitfalls of LLM-judged metrics

  • Noise: scores vary from run to run. Use temperature=0 for the judge, enough questions (50+), and compare with a tolerance, not exact equality.
  • Judge bias: judges favour longer answers, and answers in their own style. Use a different (ideally stronger) model as the judge than as the generator.
  • Calibrate against humans: have a person label 30–50 samples and check the judge agrees. If it doesn't, the metric is measuring the wrong thing.
  • Cost: each metric makes several LLM calls per sample. 200 questions × 4 metrics adds up, so cache the results.
  • Don't optimise only the metric. Read the outputs. A RAG system that answers "I don't know" to everything scores perfect faithfulness.

Interview questions

How do you evaluate a RAG system?

Separate retrieval from generation. Build an eval set of real and synthetic questions, including unanswerable ones, with reference answers and labelled relevant chunks. Measure retrieval with recall@k, MRR and context precision/recall, and generation with faithfulness and answer relevancy (RAGAS or similar). Run it in CI to catch regressions, sample live traffic with the reference-free metrics, and add production failures to the set.

Explain faithfulness and how RAGAS computes it.

Faithfulness measures whether the answer is grounded in the retrieved contexts. An LLM breaks the answer into atomic claims, then checks each claim against the contexts. The score is supported claims divided by total claims. It needs no reference answer, so it works on production traffic. It doesn't check correctness: if a wrong chunk was retrieved, a faithful answer can still be wrong.

What's the difference between context precision and context recall?

Context recall asks whether the retrieved chunks contain everything needed for the reference answer, measuring completeness. Context precision asks whether the relevant chunks are ranked at the top rather than buried under irrelevant ones, measuring ranking quality and noise. Low recall points to indexing and retrieval coverage. Low precision points to ranking, so add a reranker or send fewer chunks.

Faithfulness is high but users say answers are wrong. Why?

The model faithfully repeated the wrong context: stale documents, the wrong product version, or irrelevant chunks. Check context recall and precision, freshness filters and metadata. Also check factual correctness against references, because faithfulness only measures grounding, not truth.

Practice

  • Add a fourth retriever run to RUNS that finds everything but ranks the relevant chunks last. Which metric catches it?
  • Add an unanswerable question to EVAL_SET. What should relevant and reference be?
  • Change MAX_DROP in 7.7 to 0.07 and see whether the release passes. Is that a safe setting?

Next: 8. RAG production checklist Back to: RAG overview · Notes overview