5. Running evals in CI

Intermediate · 7 min read

A prompt tweak that fixes one complaint can quietly break ten other answers. Normal tests can't catch that, because LLM output isn't a fixed string. The answer is an eval suite that runs on every change, like unit tests, and a gate that blocks the release when quality drops. (Building eval sets and metrics is covered in Evaluation & guardrails and Evaluation with RAGAS.)

5.1 Score a candidate against the live version

Each test case has a question, what a good answer must contain, and tags so you can see which kind of question got worse:

CASES = [
    {"id": "r1", "must": ["5 working days"], "tags": ["refunds"]},
    {"id": "s1", "must": ["1 day"], "tags": ["shipping"]},
    {"id": "s2", "must": ["don't know"], "tags": ["shipping", "unanswerable"]},   # "Do you ship to Japan?"
    {"id": "w1", "must": ["not covered"], "tags": ["warranty"]},
]
ANSWERS = {
    "v7 (live)":      {"r1": "Refunds take 5 working days.", "s1": "Express takes 1 day.",
                       "s2": "I don't know based on the documents.", "w1": "Accidental damage is not covered."},
    "v8 (candidate)": {"r1": "Refunds take 5 working days.", "s1": "Express takes 1 day.",
                       "s2": "Yes, we ship worldwide including Japan.", "w1": "Accidental damage is not covered."},
}

for version, answers in ANSWERS.items():
    failed = [c["id"] for c in CASES if not all(m in answers[c["id"]].lower() for m in c["must"])]
    print(f"{version:15} {1 - len(failed) / len(CASES):.0%} passed  failed: {failed or '-'}")

Output

v7 (live)       100% passed  failed: -
v8 (candidate)  75% passed  failed: ['s2']

The candidate started answering a question the documents can't answer (shipping to Japan): a hallucination that a quick manual check could easily miss, and exactly what an eval gate exists to catch.

5.2 The gate

Compare the candidate with the baseline in production, overall and per tag, with two kinds of rule:

• No regression beyond noise: LLM scores wobble a little between runs, so allow a small margin (e.g. 3–5 points).
• Hard floors that never move: safety cases always pass; unanswerable questions are always refused.

If either rule fails, CI fails and the change doesn't ship. Run a small, fast suite on every pull request that touches prompts, retrieval or model settings, and the full suite nightly.

In the full LLMOps guide (PDF)

• A complete eval runner with rule-based scorers, per-tag reports and saved results
• The release gate in code: baselines, noise margins, hard floors and exit codes
• A ready-to-use GitHub Actions workflow (smoke suite on PRs, full suite nightly, reports as artifacts)
• Keeping CI evals cheap and stable: caching, repeats, temperature and baseline updates
• Online evals on live traffic, and turning production failures into new test cases
• Model answers to the interview questions below

See the full guide in the Store → · Get notified when it goes on sale ↓

Interview questions

• How do you test changes to prompts or models?
• How do you handle non-deterministic results in CI evals?
• Why compare per tag instead of only the overall score?

The full LLMOps guide: 36 pages, 9 chapters. Deployment, serving, caching, observability, evals in CI, cost and security, with complete code and model interview answers. Get one email when it goes on sale.

Notify me

Next: 6. Cost control Back to: LLMOps overview · Notes overview