Also known as: model distillation · teacher-student
Knowledge distillation is a training technique in which a smaller student model learns to reproduce the behaviour of a larger teacher model, yielding a compact model that retains much of the teacher's capability.
Distillation transfers more than the teacher's final answers — the teacher's full output distribution carries information about relative confidence across alternatives, which is a richer training signal than hard labels alone.
It underpins much of the current generation of small, efficient models. It also raises licensing questions, since many commercial model terms explicitly prohibit using outputs to train competing models.
Fine-Tuning
Fine-tuning is the process of further training a pre-trained model on a smaller, task-specific dataset to adapt its behaviour, style, or domain knowledge without training from scratch.
Quantization
Quantization reduces the numerical precision of a model's weights — for example from 16-bit floating point to 8-bit or 4-bit integers — shrinking memory requirements and speeding inference at some cost to output quality.
Open-Weight Model
An open-weight model is one whose trained parameters are publicly downloadable, allowing anyone to run, inspect, fine-tune, and deploy it on their own infrastructure without calling an external API.
Inference
Inference is the process of running a trained model to produce output for a given input — as distinct from training, which is the process of creating the model in the first place.