Quantization is reducing the precision of a model’s internal numbers — say FP16 down to INT4. Size and memory drop several-fold; a model needing 40 GB of RAM runs in 8. Formats like GGUF, AWQ and GPTQ are all names of this game.
The price: quality loss. With 4-bit and good calibration data the loss is near-imperceptible; with 2-bit and no care, answers turn mushy. The craft is balancing “ran on your device” against “still answers correctly.”
Network effect: quantized models make inference cheap in latency and cost — exactly what moves AI from the server to the phone.
