Also known as: Reinforcement Learning from Human Feedback · preference tuning
RLHF (Reinforcement Learning from Human Feedback) is a training technique that aligns a model's behaviour with human preferences by training a reward model on human comparisons of outputs, then optimising the language model against that reward signal.
RLHF is largely responsible for the shift from models that merely continue text to models that behave as helpful assistants. The base model supplies capability; preference tuning shapes how that capability is expressed.
Known failure modes include sycophancy, where the model learns that agreement is rated favourably, and reward hacking, where it optimises for the proxy signal rather than genuine quality. Variants such as Constitutional AI and DPO address parts of this with different trade-offs.
Fine-Tuning
Fine-tuning is the process of further training a pre-trained model on a smaller, task-specific dataset to adapt its behaviour, style, or domain knowledge without training from scratch.
Large Language Model
A large language model (LLM) is a neural network trained on vast quantities of text to predict likely continuations of a sequence, producing systems capable of generating coherent language, answering questions, and writing code.
AI Alignment
AI alignment is the field concerned with ensuring AI systems reliably pursue the goals their operators actually intend, rather than the literal specification given or some proxy the system learned during training.
AI Evaluation
AI evaluation is the practice of systematically measuring model performance against defined tasks and criteria, using benchmarks, human review, or automated graders to quantify capability and detect regressions.