Fine-tuning with LoRA and QLoRA: a decision-and-execution guide
Before you rent a GPU, decide: is the problem behavior or knowledge? This guide walks the LoRA/QLoRA path from data to deployment with the same evaluation-first logic.

Decide first: fine-tune or RAG?
Fine-tuning changes behavior, not knowledge. If the model "does not know", inject knowledge with RAG; if it "knows but answers differently" — tone, output format, domain jargon, your team's specific format — LoRA is the right tool. The screening question: given the correct answer in front of it, will it produce it? Yes means behavior problem, so fine-tune; no means knowledge problem, so RAG.
- Fresh knowledge or company docs → RAG
- Tone, format, jargon → LoRA/QLoRA
- Both needed: RAG for knowledge + light LoRA for behavior
Why LoRA and QLoRA? The economics of adaptation
Full training of a 70B model costs thousands of dollars and days. LoRA trains only the tiny low-rank matrices beside the network instead of all weights — a fraction of a percent of parameters. QLoRA goes one step further: it quantizes the base model to 4-bit and runs that same LoRA on top; result: fine-tuning a 7B model on a single consumer 24 GB GPU. The output is a few-hundred-MB adapter file, not a multi-GB model copy.
- Output = lightweight adapter, base model untouched
- QLoRA: 4-bit base + LoRA = consumer GPU
- Multiple adapters per base model, one per team
Data: the bottleneck everyone underestimates
LoRA does not need millions of samples; it needs hundreds to a couple thousand clean, consistent ones. The quality formula: ten excellent examples per real work format beat ten thousand irrelevant generated ones. The construction pattern: write real examples yourself (or generate with a frontier model and review by hand), split train/eval from day one, and never train on eval.
- 200–2,000 clean samples suffice
- Sample output format = expected format
- Split eval from day one
Execution: the one-week path with open tools
The default 2026 path: Hugging Face TRL or LLaMA-Factory for training, an open Qwen/Llama/Gemma as the base, and re-evaluation on the same golden set you built before fine-tuning. Default LoRA hyperparameters (rank 8–32, zero dropout, 2e-4 learning rate) are a fine start; but the win number comes from your eval set, not the loss curve. Deploying the adapter is one extra config line in both vLLM and Ollama.
- TRL/LLaMA-Factory + open 7–14B base
- Golden set before and after, mandatory
- Adapter on vLLM/Ollama, one config line