2. Few-shot examples in depth

Beginner → Intermediate · 12 min read

An example is worth a paragraph of instructions. When a format, a labelling scheme or a tone is hard to describe, show it. But examples are powerful in both directions: the model copies their good parts and their accidents (length, wording, label frequency). This page is about choosing them on purpose.

2.1 Zero-shot, few-shot, many-shot

Approach	What you give	Use when
Zero-shot	Instructions only	The task is common and easy to describe (summarise, translate)
Few-shot	Instructions + 2–8 examples	Custom labels, an exact output format, a house style, tricky edge cases
Many-shot	Dozens to hundreds of examples	Long-context models, hard classification; check cost and latency first
Fine-tuning	Thousands of examples, trained in	Very high volume, stable task, and few-shot isn't good enough

Start zero-shot. Add examples when an eval shows the model getting the format or the edge cases wrong.

2.2 What makes a good example set

from collections import Counter

examples = [
    ("App crashes when I upload a 12 MB PDF", "bug"),
    ("Please add dark mode", "feature_request"),
    ("I was charged twice in March", "billing"),
    ("Login button does nothing on Safari", "bug"),
    ("Export to Excel shows the wrong dates", "bug"),
    ("Can I get an invoice with my GST number?", "billing"),
]

def audit(examples: list[tuple[str, str]], labels: set[str]) -> None:
    counts = Counter(label for _, label in examples)
    lengths = [len(text.split()) for text, _ in examples]
    print("label counts:", dict(counts))
    print("missing labels:", sorted(labels - set(counts)) or "none")
    print(f"length range: {min(lengths)}–{max(lengths)} words")

audit(examples, {"bug", "feature_request", "billing", "other"})

Output

label counts: {'bug': 3, 'feature_request': 1, 'billing': 2}
missing labels: ['other']
length range: 4–9 words

The audit shows two problems: bug is half the set (the model will lean towards it), and there's no other example, so the model has never seen what "none of the above" looks like. A good set:

• covers every label, roughly balanced, including the "none / other / unsure" case;
• includes the hard cases: the ones near a boundary, the ones your users actually send;
• varies length, wording and language, so the model learns the task rather than a surface pattern;
• matches the real input format (messy, lowercase, Hinglish if that's what users write).

2.3 Format examples as a conversation

For chat models, examples work best as earlier user / assistant turns. The model sees exactly what its own reply should look like:

import json

def few_shot_messages(system: str, examples: list[tuple[str, str]], query: str) -> list[dict]:
    messages = [{"role": "system", "content": system}]
    for text, label in examples:
        messages.append({"role": "user", "content": text})
        messages.append({"role": "assistant", "content": json.dumps({"label": label})})
    messages.append({"role": "user", "content": query})
    return messages

msgs = few_shot_messages('Classify the ticket. Reply with JSON: {"label": ...}', examples[:2], "Refund not received")
for m in msgs:
    print(f"{m['role']:9} | {m['content']}")

Output

system    | Classify the ticket. Reply with JSON: {"label": ...}
user      | App crashes when I upload a 12 MB PDF
assistant | {"label": "bug"}
user      | Please add dark mode
assistant | {"label": "feature_request"}
user      | Refund not received

Alternatively, put examples inside the system prompt in <example> tags. Either works; be consistent and keep the exact output format identical in every example.

2.4 Order and recency

Models are slightly biased towards the last examples they saw and towards labels that appear often. Two cheap fixes:

• Shuffle examples (with a fixed seed, so runs are reproducible) rather than grouping all of one label together.
• Don't end with a run of the same label.

import random

def interleave(examples: list[tuple[str, str]], seed: int = 7) -> list[tuple[str, str]]:
    shuffled = examples[:]
    random.Random(seed).shuffle(shuffled)
    return shuffled

print([label for _, label in interleave(examples)])

Output

['bug', 'bug', 'billing', 'bug', 'feature_request', 'billing']

2.5 Retrieve the most similar examples per input

With a large pool of labelled examples, choose the few that are most similar to the current input. This usually beats a fixed list, because the model sees the closest precedent. In production you'd embed the examples and search a vector index; here a simple word-overlap similarity shows the idea:

import math
import re
from collections import Counter

def vec(text: str) -> Counter:
    return Counter(re.findall(r"[a-z0-9]+", text.lower()))

def cosine(a: Counter, b: Counter) -> float:
    dot = sum(a[w] * b[w] for w in a)
    return dot / (math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values())) or 1)

pool = examples + [
    ("Where is my refund? It's been 10 days", "billing"),
    ("Your app is great, thanks!", "other"),
    ("PDF upload fails with error 413", "bug"),
]

def select_examples(query: str, pool: list[tuple[str, str]], k: int = 3) -> list[tuple[str, str]]:
    q = vec(query)
    return sorted(pool, key=lambda ex: cosine(q, vec(ex[0])), reverse=True)[:k]

for text, label in select_examples("PDF upload is failing with an error", pool):
    print(f"{label:15} {text}")

Output

bug             PDF upload fails with error 413
bug             App crashes when I upload a 12 MB PDF
billing         Can I get an invoice with my GST number?

Keep one or two fixed examples for the format and the "other" case, and fill the rest dynamically.

2.6 When examples hurt

• Copying content: the model reuses facts or phrases from the examples in its answer. Use clearly fictional or varied examples, and say "the examples show the format only".
• Over-narrowing: three examples of short answers make every answer short, even when the question needs more.
• Stale labels: when the label set changes, old examples quietly teach the old scheme. Version examples with the prompt.
• Reasoning models: they often do better with a clear goal and constraints than with many worked examples; test both.

Interview questions

How do you choose few-shot examples?

Cover every label including "other", balance them, include hard boundary cases, vary length and wording, and match the real input style. With a large pool, retrieve the most similar examples per input by embedding similarity, plus one or two fixed examples for the format.

What biases can few-shot examples introduce?

Majority-label bias (frequent labels get predicted more), recency bias (the last examples weigh more), and copying surface features like length or phrasing. Balance and shuffle examples, vary them, and evaluate on a held-out set.

When would you fine-tune instead of using few-shot prompts?

When volume is high enough that long prompts cost too much, the task is stable, you have thousands of good labelled examples, and few-shot plus better instructions still miss your quality bar on the eval set.

Practice

• Add two other examples to examples and re-run audit.
• Change select_examples so it always includes one other example, whatever the query.

Next: 3. Reasoning techniques Back to: Prompt Engineering overview · Notes overview