Also known as: model red teaming · adversarial testing
AI red teaming is the practice of deliberately probing a model or AI-powered application for harmful, unsafe, or policy-violating behaviour before and after deployment.
It differs from conventional evaluation by being adversarial and open-ended: rather than measuring performance on defined tasks, red teamers actively search for inputs that break intended behaviour, including jailbreaks, prompt injection, and unexpected capability in restricted domains.
For applications rather than base models, the scope must extend beyond the model to the surrounding system — what tools it can invoke, what data it can reach, and what actions a manipulated model could actually take.
Jailbreak
A jailbreak is a prompt crafted to bypass an AI model's safety training and elicit output the model was trained to refuse, typically through roleplay framing, hypothetical scenarios, or obfuscated instructions.
Prompt Injection
Prompt injection is an attack in which malicious instructions are embedded in content an AI system processes — a web page, document, or email — causing the model to follow the attacker's instructions instead of the application's.
AI Alignment
AI alignment is the field concerned with ensuring AI systems reliably pursue the goals their operators actually intend, rather than the literal specification given or some proxy the system learned during training.
AI Evaluation
AI evaluation is the practice of systematically measuring model performance against defined tasks and criteria, using benchmarks, human review, or automated graders to quantify capability and detect regressions.
Red Team
A red team is a group that simulates realistic adversary behaviour against an organisation to test not just its technical defences but its ability to detect and respond to an intrusion in progress.