Also known as: model quantization · INT8 · 4-bit quantization
Quantization reduces the numerical precision of a model's weights — for example from 16-bit floating point to 8-bit or 4-bit integers — shrinking memory requirements and speeding inference at some cost to output quality.
Quantization is what makes running capable models on consumer hardware feasible. A model requiring substantial datacentre memory at full precision may fit comfortably on a single consumer GPU at 4-bit, with quality degradation that is often modest.
Degradation is not uniform across tasks. Quantized models frequently retain conversational fluency while losing measurable accuracy on precise reasoning, arithmetic, and code generation — so evaluation should target the actual workload rather than general benchmarks.
Inference
Inference is the process of running a trained model to produce output for a given input — as distinct from training, which is the process of creating the model in the first place.
Open-Weight Model
An open-weight model is one whose trained parameters are publicly downloadable, allowing anyone to run, inspect, fine-tune, and deploy it on their own infrastructure without calling an external API.
Edge AI
Edge AI is the practice of running AI models directly on end-user devices such as phones and laptops rather than on remote servers, keeping input data local to the device.
Knowledge Distillation
Knowledge distillation is a training technique in which a smaller student model learns to reproduce the behaviour of a larger teacher model, yielding a compact model that retains much of the teacher's capability.