5. LoRA fine-tuning an open model

Advanced · 18 min read

This page fine-tunes an open model yourself with QLoRA (page 3), using the Hugging Face stack: transformers loads the model, peft adds LoRA adapters, bitsandbytes quantizes the base to 4-bit, and trl's SFTTrainer runs the training. A 1.5B-parameter model trains on a free Colab T4 in minutes; the same code scales to 8B+ on a bigger GPU.

Versions

Written against trl 1.16, peft 0.21 and recent transformers. These libraries move fast; if an argument name fails, check the TRL docs for SFTConfig. Code here needs an NVIDIA GPU and is marked "no-run".

5.1 Setup

On Google Colab: Runtime → Change runtime type → T4 GPU. Locally, you need an NVIDIA GPU with CUDA and ideally 12 GB+ of memory.

pip install -U transformers trl peft bitsandbytes datasets accelerate

# no-run — needs an NVIDIA GPU
import torch
print(torch.cuda.is_available(), torch.cuda.get_device_name(0))
print(f"{torch.cuda.get_device_properties(0).total_memory / 1e9:.0f} GB")

Pick a base model. Start small and use an instruct (chat) version, since it already follows instructions and you only need to adjust its behaviour: for example Qwen/Qwen2.5-1.5B-Instruct, meta-llama/Llama-3.2-3B-Instruct or google/gemma-3-1b-it. Some models (Llama, Gemma) are gated: accept the licence on their Hugging Face page and run huggingface-cli login first. Check the licence allows your use.

5.2 Load the data in prompt-completion form

Convert the chat JSONL from page 2 so the assistant reply is the completion. TRL then computes the loss on the completion only, which is what we want (the model learns to write replies, not to repeat customer messages):

# no-run
from datasets import load_dataset

def to_prompt_completion(example):
    messages = example["messages"]
    return {"prompt": messages[:-1], "completion": messages[-1:]}   # everything before the reply / the reply

data = load_dataset("json", data_files={"train": "train.jsonl", "validation": "valid.jsonl"})
data = data.map(to_prompt_completion, remove_columns=["messages"])
print(data)
print(data["train"][0]["completion"])

5.3 Load the model in 4-bit and configure LoRA

# no-run
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig

BASE = "Qwen/Qwen2.5-1.5B-Instruct"

compute_dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16   # T4 has no bf16

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # the 4-bit format from the QLoRA paper
    bnb_4bit_use_double_quant=True,        # quantize the quantization constants too: saves a bit more
    bnb_4bit_compute_dtype=compute_dtype,
)

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",           # adapters on every linear layer: best quality for QLoRA
    task_type="CAUSAL_LM",
)

5.4 Train with SFTTrainer

# no-run
from trl import SFTConfig, SFTTrainer

config = SFTConfig(
    output_dir="shopkart-qwen-lora",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,         # effective batch size 16
    learning_rate=2e-4,                    # LoRA likes a higher LR than full fine-tuning
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    max_length=1024,                       # above the p95 of your examples
    gradient_checkpointing=True,           # less memory, a bit slower
    bf16=compute_dtype == torch.bfloat16,
    fp16=compute_dtype == torch.float16,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_steps=50,
    report_to="none",                      # or "wandb" / "tensorboard" for charts
)

trainer = SFTTrainer(
    model=model,
    args=config,
    train_dataset=data["train"],
    eval_dataset=data["validation"],
    processing_class=tokenizer,
    peft_config=lora,
)

trainer.model.print_trainable_parameters()   # e.g. "trainable params: 18M || all params: 1.5B || trainable%: 1.2"
trainer.train()
trainer.save_model("shopkart-qwen-lora/final")   # saves only the adapter: tens of MB

While it trains, watch loss and eval_loss in the logs. Training loss should fall steadily; eval_loss should fall and then flatten. If eval_loss starts rising, stop: use an earlier checkpoint or fewer epochs. An out-of-memory error means lowering per_device_train_batch_size (and raising gradient_accumulation_steps to compensate) or max_length.

5.5 Try the adapter

# no-run
from peft import AutoPeftModelForCausalLM

tuned = AutoPeftModelForCausalLM.from_pretrained("shopkart-qwen-lora/final", device_map="auto",
                                                 torch_dtype=compute_dtype)

messages = [{"role": "system", "content": "You are ShopKart's support assistant."},
            {"role": "user", "content": "parcel says delivered but nothing came!!"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(tuned.device)
output = tuned.generate(inputs, max_new_tokens=150, do_sample=False)
print(tokenizer.decode(output[0][inputs.shape[1]:], skip_special_tokens=True))

Compare with the base model on the same messages. Then run the proper evaluation from page 6 before you trust it: a few nice-looking samples prove very little.

5.6 Merge, share and serve

Merge the adapter into the weights for simpler serving. Load the base in 16-bit (not 4-bit) for this:

# no-run
from peft import PeftModel

base16 = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
merged = PeftModel.from_pretrained(base16, "shopkart-qwen-lora/final").merge_and_unload()
merged.save_pretrained("shopkart-qwen-merged")
tokenizer.save_pretrained("shopkart-qwen-merged")

# optional: share on the Hugging Face Hub (private by default is wise)
merged.push_to_hub("your-name/shopkart-qwen", private=True)
tokenizer.push_to_hub("your-name/shopkart-qwen", private=True)

Ways to serve it:

Option	How	Good for
vLLM	vllm serve shopkart-qwen-merged, or keep the base and add --enable-lora --lora-modules shopkart=shopkart-qwen-lora/final	Production throughput; many adapters on one base model
Ollama	Convert the merged model to GGUF with llama.cpp's convert_hf_to_gguf.py, then ollama create from a Modelfile	Laptops and on-premise CPUs
Hugging Face Inference Endpoints	Deploy the Hub repo	Managed hosting
Managed LoRA hosts (Together, Fireworks)	Upload the adapter	Pay-per-token serving of your adapter

vLLM exposes an OpenAI-compatible API, so your application code doesn't change: point the client's base_url at it (see Serving open models).

5.7 Faster options

• Unsloth wraps this same workflow with optimised kernels: typically ~2× faster and lower memory, with free Colab notebooks for most popular models.
• Axolotl and LLaMA-Factory drive the whole process from a YAML config, convenient for repeatable runs.

Whatever the tool, the steps don't change: good data → QLoRA config → watch eval loss → evaluate → merge and serve.

Interview questions

Walk me through fine-tuning an open-source LLM.

Choose a small instruct base model with a suitable licence; prepare chat data and convert it to prompt-completion so loss is on responses; load the model in 4-bit with bitsandbytes; add LoRA adapters with PEFT (rank, alpha, target modules); train with TRL's SFTTrainer while monitoring training and validation loss; evaluate against the base model on a held-out set; then merge the adapter and serve with vLLM or convert to GGUF for Ollama.

How do you avoid out-of-memory errors when fine-tuning?

Use QLoRA (4-bit base), smaller per-device batch size with gradient accumulation, gradient checkpointing, shorter max sequence length, fewer target modules or lower rank, and a smaller base model. Check memory before scaling up.

Why convert chat data to prompt-completion format?

So the trainer computes loss only on the completion (the assistant reply). Training on the full sequence would also teach the model to generate system prompts and user messages, wasting capacity and sometimes degrading behaviour.

Should you merge the LoRA adapter or keep it separate?

Merge for the simplest, fastest single-model deployment. Keep adapters separate when serving many fine-tunes on one base model (vLLM can hot-swap LoRA adapters per request), when you want to update adapters independently, or to save storage.

Practice

• Run the notebook on Colab with 100 examples and 1 epoch; note the training time and memory used.
• Compare r=8 and r=32 on the same data using eval_loss and your eval set.
• Serve the merged model with vLLM or Ollama and call it with the OpenAI SDK.

Next: 6. Evaluating and deploying Back to: Fine-Tuning overview · Notes overview