Also known as: transformer architecture · attention mechanism
The transformer is a neural network architecture, introduced in 2017, that uses self-attention to weigh the relevance of every input token against every other, and which underpins essentially all modern large language models.
Its decisive advantage over earlier recurrent architectures was parallelism. Recurrent networks processed sequences step by step, making training slow; transformers process an entire sequence simultaneously, which is what made training on internet-scale corpora economically feasible.
Self-attention lets each token's representation incorporate information from any other token regardless of distance, solving the long-range dependency problem that limited prior approaches. The cost is that attention scales quadratically with sequence length, which is why context window expansion is an active area of architectural research.
Large Language Model
A large language model (LLM) is a neural network trained on vast quantities of text to predict likely continuations of a sequence, producing systems capable of generating coherent language, answering questions, and writing code.
Token
A token is the atomic unit of text that a language model processes — typically a common word, a word fragment, or a punctuation mark — produced by splitting input text through a tokenizer before the model sees it.
Context Window
The context window is the maximum number of tokens a language model can consider at once, encompassing both the input prompt and the generated output.
Embedding
An embedding is a numerical vector representation of text, images, or other data, positioned in a high-dimensional space such that semantically similar items sit close together.
Inference
Inference is the process of running a trained model to produce output for a given input — as distinct from training, which is the process of creating the model in the first place.