Distillation is transferring knowledge from a teacher model to a smaller student: the student imitates the teacher’s outputs — not just final answers but probability distributions and reasoning. The result is a smaller model that mirrors the teacher at a fraction of the cost.
The standard 2026 chain: an expensive large teacher builds, a cheap fast student ships. The quick Gemma, Phi and Qwen versions are all distillation children of larger models.
Versus fine-tuning: distillation compresses the power of an existing model; fine-tuning targets a new domain. They are often combined — distill first, then tune.
