Skip to content

Search across models, tools, services, companies, providers, APIs, plans, articles and news.

Training & tuning

Quantization

Quantization

Quantization

Reducing the precision of a model’s internal numbers (e.g. 16-bit to 4-bit) so its size and memory needs shrink dramatically and it runs on ordinary hardware — with minor quality loss.

Quantization is reducing the precision of a model’s internal numbers — say FP16 down to INT4. Size and memory drop several-fold; a model needing 40 GB of RAM runs in 8. Formats like GGUF, AWQ and GPTQ are all names of this game.

The price: quality loss. With 4-bit and good calibration data the loss is near-imperceptible; with 2-bit and no care, answers turn mushy. The craft is balancing “ran on your device” against “still answers correctly.”

Network effect: quantized models make inference cheap in latency and cost — exactly what moves AI from the server to the phone.

Back to glossary