Local deployment: Ollama and llama.cpp on home hardware
Today 3–14B models run on ordinary laptops. This guide covers tool, model, quantization and settings choices for fast offline chat on your 8–32 GB of RAM.

Why local? Privacy, latency and zero marginal cost
Three real reasons: data never leaves the device, network and API-queue latency disappear, and the bill stays flat no matter the volume. The trade-off: a local model trails the cloud frontier. Its right place in hybrid architecture: sensitive and routine work local, hard work cloud.
- Sensitive + routine high-volume work = local
- A 3–8B model suffices for many tasks
- Hybrid local+cloud beats either alone
Tool choice: Ollama or raw llama.cpp
llama.cpp is the GGUF runtime engine; Ollama packages that engine with model serving, an OpenAI-compatible API and model management. For daily use and tool integrations Ollama is the default; for maximum squeezing on niche hardware or embedding into your own binary, run llama.cpp raw.
- Ollama: ready service + OpenAI-compatible API
- llama.cpp: full control for custom deployments
- GGUF is the shared format
The RAM rule: quantized models fit your machine
Fitting formula: Q4 runs at roughly half of FP16; a 7B at Q4 is about 5 GB RAM, 14B about 10 GB, 32B about 20 GB. 8 GB RAM means Qwen/Llama 3B; 16 GB means 7–8B at Q4; 32 GB means a comfortable 14B. Rule of thumb: always start at Q4_K_M and only upgrade on evidence of quality loss on your own eval.
- Q4 ≈ half of FP16; start at Q4_K_M
- 8 GB RAM → 3B; 16 GB → 7B; 32 GB → 14B
- Long local context eats RAM