Also known as: tokenization · tokenizer · BPE
A token is the atomic unit of text that a language model processes — typically a common word, a word fragment, or a punctuation mark — produced by splitting input text through a tokenizer before the model sees it.
Tokens are not words. Common words are usually single tokens while rare words fragment into several, and a rough English heuristic is that a token averages about four characters. Non-English text and code frequently tokenize far less efficiently, which has real cost and context-length consequences.
Tokenization explains several otherwise puzzling model behaviours, including difficulty with character-level tasks such as counting letters in a word or reversing strings — the model never sees individual characters, only token identifiers.
Large Language Model
A large language model (LLM) is a neural network trained on vast quantities of text to predict likely continuations of a sequence, producing systems capable of generating coherent language, answering questions, and writing code.
Context Window
The context window is the maximum number of tokens a language model can consider at once, encompassing both the input prompt and the generated output.
Transformer
The transformer is a neural network architecture, introduced in 2017, that uses self-attention to weigh the relevance of every input token against every other, and which underpins essentially all modern large language models.
Inference
Inference is the process of running a trained model to produce output for a given input — as distinct from training, which is the process of creating the model in the first place.