3. How fine-tuning works: LoRA and QLoRA¶
Intermediate → Advanced · 16 min read
You can fine-tune through an API without knowing any of this (page 4). But the moment you train an open model yourself, or an interviewer asks "why LoRA?", you need the picture: what the model is optimising, why it needs so much memory, and how LoRA and QLoRA cut that memory by 10× or more.
3.1 What training optimises¶
A language model predicts the next token. Fine-tuning shows it your examples and nudges the weights so the correct next tokens in your assistant replies become more likely. The measure of "how wrong" is the cross-entropy loss: the average of −log(probability the model gave to the right token).
Only the assistant's tokens should count. The system prompt and user message are context, not something to learn to write. Training code does this with a loss mask:
import math
# token, whose turn it belongs to, and the probability the model gave to that correct token
tokens = [("You", "system", 0.20), ("are", "system", 0.30), ("support", "system", 0.10),
("Where", "user", 0.05), ("is", "user", 0.40), ("A100", "user", 0.01),
("[delivery]", "assistant", 0.30), ("Checking", "assistant", 0.20), ("now", "assistant", 0.60)]
def loss(rows):
return sum(-math.log(p) for _, _, p in rows) / len(rows)
assistant_only = [t for t in tokens if t[1] == "assistant"]
print(f"loss on all tokens: {loss(tokens):.2f}")
print(f"loss on assistant tokens: {loss(assistant_only):.2f}")
# after some training the model is more confident about the assistant's tokens
trained = [("[delivery]", "assistant", 0.90), ("Checking", "assistant", 0.70), ("now", "assistant", 0.85)]
print(f"after training: {loss(trained):.2f}")
Loss going down on the training set only means the model is memorising your examples well. What matters is the loss on the validation set, and your eval scores (page 6).
3.2 The hyperparameters that matter¶
| Setting | What it controls | Typical starting point |
|---|---|---|
| Epochs | How many times the model sees every example | 1-3. More examples → fewer epochs |
| Learning rate | Size of each weight update | Full FT: ~1e-5. LoRA: ~1e-4 to 2e-4 |
| Batch size (× gradient accumulation) | Examples per update | Effective batch of 8-32 |
| Max length | Longest sequence; longer ones are truncated | Above your data's p95 length |
| Warmup / scheduler | Ramp the learning rate up, then down | ~3-5% warmup, cosine or linear decay |
Reading the loss curves:
loss
│ ╲ training loss keeps falling...
│ ╲___
│ ╲____________________ train
│ ╲__ ___---- validation ...validation loss turns up: overfitting.
│ ╲______----‾‾‾ Stop at the lowest point (fewer epochs).
└──────────────────────────────── steps
Overfitting (validation loss rising, model parroting training replies) means too many epochs or too little data. Catastrophic forgetting (the model gets worse at general tasks it used to do) means the learning rate or epochs are too high, or the data is too narrow. LoRA reduces both risks, which is part of why it's the default.
3.3 Why full fine-tuning needs so much memory¶
Updating every weight needs memory for the weights, their gradients, and the optimizer's bookkeeping (Adam keeps two extra numbers per weight), plus activations:
def full_finetune_gb(params_billion: float) -> dict:
p = params_billion * 1e9
return {
"weights (bf16, 2 B)": p * 2 / 1e9,
"gradients (bf16, 2 B)": p * 2 / 1e9,
"Adam states (fp32, 8 B)": p * 8 / 1e9,
"fp32 master weights (4 B)": p * 4 / 1e9,
}
for size in (1, 8, 70):
parts = full_finetune_gb(size)
print(f"{size:>2}B model: ~{sum(parts.values()):,.0f} GB before activations")
print(full_finetune_gb(8))
1B model: ~16 GB before activations
8B model: ~128 GB before activations
70B model: ~1,120 GB before activations
{'weights (bf16, 2 B)': 16.0, 'gradients (bf16, 2 B)': 16.0, 'Adam states (fp32, 8 B)': 64.0, 'fp32 master weights (4 B)': 32.0}
About 16 bytes per parameter: an 8B model needs well over 100 GB, several data-centre GPUs. And every fine-tune produces a complete new copy of the model to store and serve. That's the problem LoRA solves.
3.4 LoRA: train a small add-on instead¶
LoRA (Low-Rank Adaptation) freezes the original weights and learns a small correction for some layers.
For a weight matrix W (say 4096 × 4096), instead of learning a full 4096 × 4096 update, it learns two thin
matrices B (4096 × r) and A (r × 4096), with a small rank r like 16. The layer's output becomes:
output = W·x + (alpha / r) · B·A·x
import numpy as np
d, r = 4096, 16
full_update = d * d
lora_params = d * r + r * d
print(f"full update: {full_update:,} params LoRA r={r}: {lora_params:,} params "
f"({lora_params / full_update:.2%})")
rng = np.random.default_rng(0)
d_small, r_small, alpha = 8, 2, 4
W = rng.normal(size=(d_small, d_small)) # frozen pre-trained weights
A = rng.normal(size=(r_small, d_small)) * 0.1 # trained (random init)
B = np.zeros((d_small, r_small)) # trained (zero init: the model starts unchanged)
x = rng.normal(size=d_small)
def lora_forward(x):
return W @ x + (alpha / r_small) * (B @ (A @ x))
print("unchanged at start:", np.allclose(lora_forward(x), W @ x))
B = rng.normal(size=(d_small, r_small)) * 0.1 # pretend training has updated B
delta = (alpha / r_small) * (B @ A)
print("rank of the learned update:", np.linalg.matrix_rank(delta))
W_merged = W + delta # merge for deployment: no extra cost at inference
print("merged == adapter:", np.allclose(W_merged @ x, lora_forward(x)))
full update: 16,777,216 params LoRA r=16: 131,072 params (0.78%)
unchanged at start: True
rank of the learned update: 2
merged == adapter: True
What this buys you:
- Under 1% of the parameters are trained, so gradients and optimizer states are tiny. Memory is dominated by the frozen weights.
Bstarts at zero, so training starts from exactly the original model.- The result is an adapter of a few MB to a few hundred MB, not a new multi-GB model. You can keep one base model and swap adapters per customer or task.
- For deployment,
W + B·Acan be merged into the weights, so there's no inference slowdown.
It works because the change fine-tuning needs is, in practice, low-rank: you're steering behaviour, not rebuilding the model's knowledge.
LoRA settings¶
| Setting | Meaning | Common values |
|---|---|---|
r (rank) |
Size of the adapter: capacity to learn | 8-64. 16 is a good default |
lora_alpha |
Scale of the update (effective scale = alpha / r) | Often equal to r or 2r |
target_modules |
Which layers get adapters | Attention projections (q_proj, k_proj, v_proj, o_proj), or "all-linear" for best quality |
lora_dropout |
Regularisation | 0-0.1 |
Higher rank and more target modules mean more capacity and memory. For style and format tasks, small ranks are enough; adapting to a very different domain benefits from higher rank and all linear layers.
3.5 QLoRA: a 4-bit frozen base¶
With LoRA, the frozen weights are now the biggest memory cost. QLoRA stores them in 4-bit (the NF4 format) instead of 16-bit, while the small LoRA adapters still train in 16-bit:
def lora_memory_gb(params_billion: float, bits: int, trainable_fraction: float = 0.005) -> float:
p = params_billion * 1e9
frozen = p * bits / 8
trainable = p * trainable_fraction * (2 + 2 + 8 + 4) # weights, grads, Adam, master copy
activations_and_overhead = 2e9 # rough, grows with batch size and length
return (frozen + trainable + activations_and_overhead) / 1e9
print(f"{'model':>6} {'LoRA (16-bit)':>14} {'QLoRA (4-bit)':>14}")
for size in (1, 3, 8, 14, 32, 70):
print(f"{size:>5}B {lora_memory_gb(size, 16):>12.0f} GB {lora_memory_gb(size, 4):>12.0f} GB")
model LoRA (16-bit) QLoRA (4-bit)
1B 4 GB 3 GB
3B 8 GB 4 GB
8B 19 GB 7 GB
14B 31 GB 10 GB
32B 69 GB 21 GB
70B 148 GB 43 GB
So an 8B model that needs >120 GB to fully fine-tune can be QLoRA-trained in roughly 7-10 GB: a free Colab T4 (16 GB), a single RTX 4090 (24 GB), or a modest cloud GPU. The quality is close to 16-bit LoRA for most tasks; training is somewhat slower because the weights are de-quantized on the fly.
| GPU memory | What you can comfortably fine-tune |
|---|---|
| 16 GB (Colab T4, RTX 4060 Ti 16 GB) | QLoRA up to ~8B |
| 24 GB (RTX 4090, L4, A10) | QLoRA up to ~14B; LoRA up to ~8B |
| 48-80 GB (A6000, A100, H100) | QLoRA up to ~70B; LoRA up to ~32B |
(Long sequences and big batches need more; gradient checkpointing trades speed for memory.)
Interview questions¶
Explain LoRA.
LoRA freezes the pre-trained weights and adds a trainable low-rank update to selected layers: W + (alpha/r)·B·A, where B and A are thin matrices of rank r. Only A and B are trained (often under 1% of parameters), which cuts memory and storage dramatically, starts from the unchanged model (B is initialised to zero), and produces small adapters that can be swapped or merged into the weights for inference with no extra latency.
What's the difference between LoRA and QLoRA?
QLoRA is LoRA on top of a base model quantized to 4-bit (NF4). The frozen weights take a quarter of the memory, while adapters train in 16-bit. It lets you fine-tune much larger models on a single GPU with quality close to 16-bit LoRA, at some cost in training speed.
Why does full fine-tuning need so much GPU memory?
Besides the weights, training stores gradients and optimizer states for every parameter (Adam keeps two moments, often in fp32) and usually an fp32 master copy: roughly 16 bytes per parameter, plus activations. An 8B model needs well over 100 GB.
How do you know if a fine-tuned model is overfitting?
Training loss keeps falling while validation loss starts rising; outputs start copying training replies word for word; eval scores on held-out data drop. Fixes: fewer epochs, a lower learning rate, more varied data, or a smaller LoRA rank.
Which loss is used in supervised fine-tuning, and on which tokens?
Cross-entropy (negative log-likelihood) of the correct next token, averaged over the tokens being learned. The prompt tokens are usually masked out so only the assistant's response contributes, because the model should learn to write responses, not to reproduce prompts.
Practice¶
- Change
rin 3.4 to 4, 8 and 64 and compare the parameter counts for a 4096-wide layer. - Use the QLoRA calculator to find the largest model you could fine-tune on your own GPU (or a free T4).
- In 3.1, add more user tokens with low probabilities and check that the assistant-only loss doesn't change.
Next: 4. Fine-tuning with an API Back to: Fine-Tuning overview · Notes overview