Also known as: alignment · AI safety
AI alignment is the field concerned with ensuring AI systems reliably pursue the goals their operators actually intend, rather than the literal specification given or some proxy the system learned during training.
The core difficulty is specification: writing down what we want precisely enough that optimising for it produces the intended behaviour. Systems optimise the objective they are given, which is frequently not the objective their designers meant.
Alignment research spans immediate practical concerns — refusal behaviour, jailbreak resistance, honesty about uncertainty — and longer-horizon questions about oversight of systems that exceed human capability in a domain. The two are related but not identical, and conflating them muddles most public discussion.
RLHF
RLHF (Reinforcement Learning from Human Feedback) is a training technique that aligns a model's behaviour with human preferences by training a reward model on human comparisons of outputs, then optimising the language model against that reward signal.
AI Evaluation
AI evaluation is the practice of systematically measuring model performance against defined tasks and criteria, using benchmarks, human review, or automated graders to quantify capability and detect regressions.
Prompt Injection
Prompt injection is an attack in which malicious instructions are embedded in content an AI system processes — a web page, document, or email — causing the model to follow the attacker's instructions instead of the application's.
Jailbreak
A jailbreak is a prompt crafted to bypass an AI model's safety training and elicit output the model was trained to refuse, typically through roleplay framing, hypothetical scenarios, or obfuscated instructions.
AI Governance
AI governance refers to the policies, regulations, and internal controls that determine how AI systems may be developed and deployed, including risk assessment, transparency obligations, and accountability for harms.