1. When to fine-tune¶
Intermediate · 12 min read
"Should we fine-tune a model on our data?" comes up in almost every GenAI project, and the right answer is usually "not yet". Fine-tuning is powerful but slow to iterate, needs good data, and solves a narrower problem than most people think. This page is about knowing when it's the right tool.
1.1 What fine-tuning actually does¶
Fine-tuning means continuing to train a pre-trained model on your own examples, so its weights shift toward the behaviour in those examples.
| Fine-tuning is good at teaching | Fine-tuning is bad at teaching |
|---|---|
| A format: always this JSON, this report layout | Facts that change (prices, policies, stock) |
| A style or tone: your brand voice, terse clinical notes | Private knowledge you need cited (use RAG) |
| A narrow task done consistently: classify into your 40 categories | Things the base model fundamentally can't do |
| Shorter prompts: behaviour baked in instead of 3 pages of instructions | Anything you need to update tomorrow |
| Smaller, cheaper models matching a big model on one task (distillation) | Fixing a broken retrieval pipeline |
The key intuition: fine-tuning changes how a model behaves, RAG changes what it knows at answer time. A model fine-tuned on your policy documents may sound like your company, but it will still misquote the refund window, and it can't show you which page the answer came from.
1.2 Prompting vs RAG vs fine-tuning¶
flowchart TD
S[Start: the model doesn't do what you want] --> P{Tried a better prompt?<br/>instructions, examples, structure}
P -->|No| P1[Prompt engineering first]
P -->|Yes, still failing| K{Is the problem missing<br/>or changing knowledge?}
K -->|Yes| R[RAG: retrieve the facts at query time]
K -->|No: format, style, consistency, cost| D{Do you have hundreds of<br/>good examples and an eval?}
D -->|No| E[Build the eval set and collect examples first]
D -->|Yes| F[Fine-tune]| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| Changes | The instructions | What the model can see | The model's weights |
| Best for | Most tasks; fast iteration | Knowledge, freshness, citations | Format, style, narrow skills, cost |
| Time to change | Minutes | Hours (re-index) | Hours to days (new data, retrain, re-evaluate) |
| Data needed | A few examples | Your documents | Hundreds to thousands of curated examples |
| Upfront cost | None | Indexing, vector DB | Training compute or API fees |
| Per-request cost | Can be high (long prompts) | Retrieved context adds tokens | Can be lowest (short prompts, smaller model) |
| Explainability | Read the prompt | Show the sources | Opaque |
They combine well: the most common production pattern is RAG with a fine-tuned model, where retrieval supplies the facts and fine-tuning makes the model reliably follow your answer format and cite properly.
1.3 Good and bad reasons to fine-tune¶
Good reasons:
- A prompt with 15 rules and 10 examples still gets the format wrong 5% of the time, and that 5% matters.
- You pay for a 4,000-token system prompt on every request, and a fine-tuned model could do the same with 200.
- A large model does the task well, and you want a small model to do it 10× cheaper and faster (distillation).
- A domain-specific style (medical discharge summaries, legal clause tagging) that's hard to describe but easy to show.
- You must run on-premise with a small open model, and it needs to be good at one job.
Bad reasons:
- "So the model knows our documents." Use RAG.
- "The model hallucinates." Fine-tuning on facts often makes it hallucinate more confidently. Fix retrieval and grounding.
- "We haven't tried prompting properly." Do that first; it's cheaper by orders of magnitude.
- "We have 30 examples." Not enough to beat good few-shot prompting.
- "It'll be our moat." Model generations change every few months; your data and evals are the asset.
1.4 Kinds of fine-tuning¶
| Kind | Training data | Teaches | Typical use |
|---|---|---|---|
| Supervised fine-tuning (SFT) | Prompt → ideal response pairs | "Answer like this" | Format, style, task skills. The default |
| Preference tuning (DPO, RLHF) | Prompt → chosen vs rejected response | "This answer is better than that one" | Tone, helpfulness, safety; polishing after SFT |
| Distillation | Prompts → a strong model's answers | Copy a big model's behaviour into a small one | Cost and latency reduction |
| Reinforcement fine-tuning | Prompts + a grader that scores answers | Reasoning on tasks with checkable answers | Maths, code, extraction with verifiable output |
| Continued pre-training | Large raw domain text | Domain language and vocabulary | Rare: specialised languages or jargon-heavy fields |
Most teams only ever need SFT, sometimes followed by DPO. Pages 2-6 focus on those.
1.5 Cost and break-even¶
Fine-tuning costs money up front and saves money per request (shorter prompts, a smaller model). A quick break-even check:
def monthly_cost(requests_per_day, input_tokens, output_tokens, price_in, price_out):
"""Prices in USD per 1M tokens."""
per_request = input_tokens / 1e6 * price_in + output_tokens / 1e6 * price_out
return per_request * requests_per_day * 30
# Option A: large model with a long prompt (rules + 10 few-shot examples)
big = monthly_cost(20_000, input_tokens=4_500, output_tokens=300, price_in=2.00, price_out=10.00)
# Option B: small fine-tuned model, short prompt; one-off training cost
small = monthly_cost(20_000, input_tokens=400, output_tokens=300, price_in=0.30, price_out=1.20)
training_once = 150 # data prep with a strong model + training run, USD
print(f"Large model + long prompt: ${big:,.0f}/month")
print(f"Small fine-tuned model: ${small:,.0f}/month (+ ${training_once} once)")
print(f"Pays for itself in {training_once / ((big - small) / 30):.1f} days")
Large model + long prompt: $7,200/month
Small fine-tuned model: $288/month (+ $150 once)
Pays for itself in 0.7 days
The numbers look compelling, but add the hidden costs: building the dataset, evaluation, retraining when the base model is deprecated or your requirements change, and serving (for open models). Fine-tune when the saving or quality gain is clear after those.
1.6 Decision checklist¶
- A strong prompt (clear instructions, examples, structured output) has been tried and measured.
- The problem is behaviour, format, style or cost, not missing knowledge.
- You have an evaluation set that shows the current failure rate.
- You can get at least a few hundred high-quality examples (or generate and review them).
- Someone owns retraining when the base model or the requirements change.
If any box is unticked, fix that first.
Interview questions¶
RAG or fine-tuning: how do you decide?
Ask what's missing. If the model lacks knowledge, especially private, changing or must-be-cited knowledge, use RAG. If it has the knowledge but doesn't behave right (format, tone, consistency on a narrow task) or is too expensive, consider fine-tuning. Always try prompting first, and often combine them: RAG for facts, a fine-tuned model for reliable formatting and grounding behaviour.
Can you fine-tune a model to learn your company's documents?
You can, but it's usually the wrong approach. Fine-tuning absorbs knowledge unreliably, can't be updated quickly, can't cite sources, and can increase confident hallucinations. RAG retrieves the exact text at query time and shows sources. Fine-tuning helps with how the model uses retrieved context, not with storing it.
What are the types of fine-tuning?
Supervised fine-tuning on prompt-response pairs (the default); preference tuning such as DPO or RLHF on chosen vs rejected answers; distillation from a stronger model's outputs; reinforcement fine-tuning with a grader on verifiable tasks; and continued pre-training on raw domain text. Parameter-efficient methods like LoRA can be used with most of them.
When is fine-tuning worth the cost?
When it removes a measurable failure that prompting couldn't, or when a small fine-tuned model can replace a large model or a very long prompt at high volume, and the savings outweigh data preparation, training, evaluation and retraining costs over the model's lifetime.
Practice¶
- Run the break-even calculation with your own project's volume and prices.
- List three tasks in your work and decide for each: prompt, RAG, fine-tune or a combination. Why?
- Find one task where you'd need fine-tuning, and write down how you'd measure success before starting.
Next: 2. Preparing training data Back to: Fine-Tuning overview · Notes overview