Fine-Tuning¶

Intermediate → Advanced · 6 topics

After prompting and RAG, fine-tuning is the third way to make an LLM do what you need, and the one most often used for the wrong reasons. This series starts with when to fine-tune (and when not to), then goes through the whole workflow: building the dataset, understanding LoRA and QLoRA, fine-tuning through an API and on your own GPU, and proving the result is actually better before you ship it.

# Pages 1, 2, 3 and 6 have runnable examples (standard library and NumPy).
# Page 4 needs an OpenAI API key; page 5 needs an NVIDIA GPU (a free Colab T4 works):
pip install openai                                                   # page 4
pip install transformers trl peft bitsandbytes datasets accelerate   # page 5

#	Topic	Sub-topics
1	When to fine-tune	What fine-tuning changes · prompting vs RAG vs fine-tuning · good and bad reasons · SFT, DPO, distillation · cost break-even
2	Preparing training data	Chat JSONL · how many examples · good targets · validation · deduplication · PII · splits · token lengths · synthetic data
3	How fine-tuning works: LoRA and QLoRA	Loss on assistant tokens · hyperparameters · overfitting · GPU memory · LoRA in NumPy · rank and alpha · QLoRA 4-bit
4	Fine-tuning with an API	Cost estimate · upload · supervised jobs · monitoring and checkpoints · using the model · DPO · managed options
5	LoRA fine-tuning an open model	Colab setup · prompt-completion data · 4-bit loading · `LoraConfig` · `SFTTrainer` · merge · vLLM and Ollama
6	Evaluating and deploying	The right baseline · eval harness · per-category results · LLM-as-judge · regressions · shadow and A/B · retraining

flowchart LR
    Q{Need it?} -->|prompting and RAG<br/>aren't enough| D[Data<br/>JSONL, clean, split]
    D --> T{Train}
    T -->|managed| API[Provider API<br/>SFT / DPO]
    T -->|self-hosted| L[QLoRA<br/>TRL + PEFT]
    API --> E[Evaluate vs best prompt]
    L --> E
    E --> S[Shadow → A/B → rollout]
    S --> M[Monitor] --> D

Before you start

These pages assume Transformers (what the model computes), LLMs (calling models, structured output) and the RAG series, since "RAG or fine-tuning?" is the first question you'll need to answer.

Back to: Notes overview