Skip to content

6. Evaluating and deploying

Intermediate → Advanced · 14 min read

A falling training loss says the model learned your examples. It doesn't say the model is better at the job. Fine-tuned models regularly look great on five hand-picked prompts and then fail on real traffic, forget general skills, or lose safety behaviour. This page is how you know, and how you ship it safely.

6.1 Compare against the right baseline

The question isn't "is the fine-tuned model good?". It's "is it better than the base model with our best prompt?", because that's the alternative you'd ship instead. Always evaluate three candidates on the same held-out test set (page 2):

Candidate Why it's in the comparison
Base model, simple prompt How hard the task is
Base model, best prompt (rules + few-shot examples) The real competitor
Fine-tuned model, short prompt The thing you want to ship

6.2 An evaluation harness

For the ShopKart support task, "good" is measurable: the right category tag, under 80 words, no forbidden promises. Here are the replies each candidate produced for the same test messages:

import re
from collections import defaultdict

test_set = [
    {"message": "My order A100 arrived with a cracked screen", "category": "[damaged]"},
    {"message": "parcel says delivered but nothing came!!", "category": "[delivery]"},
    {"message": "I want my money back for order A207", "category": "[refund]"},
    {"message": "do you sell gift cards?", "category": "[other]"},
    {"message": "box was crushed and the mug is broken", "category": "[damaged]"},
    {"message": "it's been 9 days, where is my stuff", "category": "[delivery]"},
]

replies = {
    "base + best prompt": [
        "[damaged]\nSo sorry! Please upload a photo under Orders > A100 > Report issue.",
        "[delivery]\nSorry! Could you share your order ID so I can check with the courier?",
        "Refund: I've processed your refund for A207, it will arrive in 3 days.",
        "[other]\nYes! Gift cards are available from Rs 500 under Gifts.",
        "[delivery]\nSorry to hear that. Let me check the delivery status for you.",
        "[delivery]\nApologies for the delay! Please share your order ID.",
    ],
    "fine-tuned": [
        "[damaged]\nSorry about the cracked screen! Upload a photo in Orders > A100 > Report issue.",
        "[delivery]\nSorry! Please share your order ID and I'll raise a courier investigation.",
        "[refund]\nI understand. I've started a refund review for A207; you'll hear back within 48 hours.",
        "[other]\nYes, gift cards from Rs 500 are under Gifts > Cards.",
        "[damaged]\nSorry about the broken mug! Please upload a photo and we'll arrange a replacement.",
        "[delivery]\nSorry for the delay! Share your order ID and I'll check right away.",
    ],
}

FORBIDDEN = [r"\bprocessed your refund\b", r"\brefund (is|has been) (approved|processed)\b"]

def score(reply: str, expected: str) -> dict:
    first_line = reply.split("\n")[0]
    return {
        "format": bool(re.fullmatch(r"\[[a-z]+\]", first_line)),
        "category": first_line == expected,
        "length": len(reply.split()) <= 80,
        "policy": not any(re.search(p, reply, re.I) for p in FORBIDDEN),
    }

for name, outputs in replies.items():
    scores = [score(r, t["category"]) for r, t in zip(outputs, test_set)]
    summary = {k: sum(s[k] for s in scores) / len(scores) for k in scores[0]}
    print(f"{name:20}", "  ".join(f"{k} {v:.0%}" for k, v in summary.items()))
Output
base + best prompt   format 83%  category 67%  length 100%  policy 83%
fine-tuned           format 100%  category 100%  length 100%  policy 100%

The fine-tuned model wins on every check, but six examples prove nothing statistically. Real test sets need 100+ examples, and differences of a few points need even more before you believe them.

Break results down

An overall score hides where the model fails. Look per category:

def per_category(outputs):
    totals, correct = defaultdict(int), defaultdict(int)
    for reply, t in zip(outputs, test_set):
        totals[t["category"]] += 1
        correct[t["category"]] += score(reply, t["category"])["category"]
    return {c: f"{correct[c]}/{totals[c]}" for c in sorted(totals)}

for name, outputs in replies.items():
    print(f"{name:20}", per_category(outputs))
Output
base + best prompt   {'[damaged]': '1/2', '[delivery]': '2/2', '[other]': '1/1', '[refund]': '0/1'}
fine-tuned           {'[damaged]': '2/2', '[delivery]': '2/2', '[other]': '1/1', '[refund]': '1/1'}

The base model's misses cluster in [damaged] and [refund]: exactly the cases to add more training examples or prompt guidance for.

6.3 Open-ended quality: LLM-as-judge

Rules can't judge tone or helpfulness. Use a strong model as a judge with a clear rubric, compare the two candidates pairwise on each test message, and randomise the order to avoid position bias:

# no-run — needs ANTHROPIC_API_KEY
import random
import anthropic

claude = anthropic.Anthropic()
RUBRIC = """Which reply is better ShopKart support? Judge: correct category, empathy, a concrete next step,
under 80 words, and no promise that a refund is approved. Answer with exactly "A" or "B"."""

def judge(message: str, reply_1: str, reply_2: str) -> int:
    swapped = random.random() < 0.5
    a, b = (reply_2, reply_1) if swapped else (reply_1, reply_2)
    verdict = claude.messages.create(
        model="claude-opus-5-5", max_tokens=16,
        system=RUBRIC,
        messages=[{"role": "user", "content": f"Customer: {message}\n\nReply A:\n{a}\n\nReply B:\n{b}"}],
    ).content[0].text.strip()
    winner_is_a = verdict.startswith("A")
    return 1 if winner_is_a != swapped else 2          # which original reply won

Check the judge against your own judgment on 30-50 cases before trusting it at scale. See Evaluation & guardrails for more.

6.4 Check for regressions

A fine-tuned model can get better at your task and worse at everything else. Before shipping, also test:

Check How Red flag
General ability (forgetting) A small set of unrelated tasks: summarise, answer a general question, follow a multi-step instruction Noticeably worse than the base model
Out-of-scope requests Messages your data never covered Forces them into one of your categories, or rambles
Safety Prompt injection, abusive users, requests for other customers' data Safety behaviour weaker than the base model
Memorisation Prompts that start like training examples Repeats training replies or any PII verbatim
Robustness Typos, Hinglish, very long messages, empty messages Format breaks

Fine-tuning can weaken the base model's safety training, even with harmless data. Run a fixed set of adversarial prompts against every new version.

6.5 Ship it carefully

flowchart LR
    E[Offline eval<br/>beats best prompt?] --> S[Shadow mode<br/>runs on real traffic,<br/>answers not shown]
    S --> AB[A/B test<br/>5-10% of users]
    AB --> F[Full rollout]
    F --> M[Monitor] --> R[Collect failures] --> T[Retrain] --> E
  • Shadow mode: send real requests to both models, show users the old one, compare offline. Catches problems the test set missed, with no user impact.
  • A/B test on a small slice, measuring what the business cares about: resolution rate, escalations, thumbs up/down, cost.
  • Version everything: the dataset (with a hash), training config, base model version and the fine-tuned model ID. Keep the previous model one config change away for instant rollback.
  • Monitor the same metrics as your eval in production: format errors, category distribution, refusals, length, latency, cost per request.

6.6 Plan for retraining

A fine-tuned model is a living asset, not a one-off:

  • New failure modes appear in production: add them (corrected) to the training data and to the test set.
  • Requirements change: a new product category, a new refund policy. With fine-tuning, that means new data and a new training run, which is why changing facts belong in RAG or the prompt, not in the weights.
  • Base models are retired: managed providers deprecate fine-tunable models, and better base models arrive every few months. Your durable assets are the dataset, eval set and pipeline, so a retrain on a new base is a routine job, not a project.

Interview questions

How do you evaluate a fine-tuned model?

On a fixed, held-out test set of real examples, compared against the base model with the best prompt you can write (not just a weak prompt). Use task metrics and rule checks where possible, LLM-as-judge with a rubric and randomised order for open-ended quality (validated against human labels), break results down by category, and run regression checks for general ability, out-of-scope requests, safety and memorisation.

What is catastrophic forgetting and how do you detect it?

The model losing general capabilities it had before fine-tuning because training pushed its weights toward a narrow task. Detect it with a small benchmark of unrelated tasks run on the base and fine-tuned models. Reduce it with LoRA, fewer epochs, a lower learning rate, and mixing some general examples into the training data.

How would you roll out a fine-tuned model to production?

Offline evaluation against the current system first, then shadow mode on real traffic, then an A/B test on a small percentage of users measuring business metrics, then gradual rollout with monitoring and a one-step rollback. Version the data, config and model, and feed production failures back into the next training round.

Your fine-tuned model scores well offline but users complain. Why?

The test set doesn't match real traffic (too clean, synthetic, or from a different time), the production system prompt differs from training, real inputs are longer or messier, or the metric doesn't capture what users care about. Sample and label real production conversations, add them to the test set, and re-evaluate.

Practice

  • Add a reply to the base model's list that breaks the 80-word limit and check the length score drops.
  • Write ten out-of-scope and adversarial test messages for your own fine-tuning task.
  • Sketch the version record you'd store for each model: dataset hash, base model, config, eval scores, date.

Back to: Fine-Tuning overview · Notes overview