Skip to content

Search across models, tools, services, companies, providers, APIs, plans, articles and news.

Training & tuning

Distillation

Distillation

Distillation

Transferring knowledge from a large model to a small one: the small model learns from the large model’s outputs to behave similarly at much lower cost.

Distillation is transferring knowledge from a teacher model to a smaller student: the student imitates the teacher’s outputs — not just final answers but probability distributions and reasoning. The result is a smaller model that mirrors the teacher at a fraction of the cost.

The standard 2026 chain: an expensive large teacher builds, a cheap fast student ships. The quick Gemma, Phi and Qwen versions are all distillation children of larger models.

Versus fine-tuning: distillation compresses the power of an existing model; fine-tuning targets a new domain. They are often combined — distill first, then tune.

Back to glossary