Also known as: model inference · serving · generation
Inference is the process of running a trained model to produce output for a given input — as distinct from training, which is the process of creating the model in the first place.
Although training dominates public discussion of AI compute, inference dominates lifetime cost for any successfully deployed model, since it runs continuously for every user request while training happens once.
Inference performance is usually memory-bandwidth-bound rather than compute-bound for language models, because generating each token requires reading the entire model's weights. This is why techniques that reduce memory traffic — quantization, batching, KV caching — deliver such large practical speedups.
Quantization
Quantization reduces the numerical precision of a model's weights — for example from 16-bit floating point to 8-bit or 4-bit integers — shrinking memory requirements and speeding inference at some cost to output quality.
Open-Weight Model
An open-weight model is one whose trained parameters are publicly downloadable, allowing anyone to run, inspect, fine-tune, and deploy it on their own infrastructure without calling an external API.
Token
A token is the atomic unit of text that a language model processes — typically a common word, a word fragment, or a punctuation mark — produced by splitting input text through a tokenizer before the model sees it.
Large Language Model
A large language model (LLM) is a neural network trained on vast quantities of text to predict likely continuations of a sequence, producing systems capable of generating coherent language, answering questions, and writing code.
Edge AI
Edge AI is the practice of running AI models directly on end-user devices such as phones and laptops rather than on remote servers, keeping input data local to the device.