Alignment and dangerous-capability evals — reading the safety science

Lesson 4 of 5 in Compute Governance, Export Controls, and the Science of AI Safety.

Every frontier-governance document you will ever review — a lab’s safety framework, an EU model report, a safety-institute evaluation — ultimately rests on claims from a young and unfinished science. An expert cannot outsource judgment about those claims. This lesson gives you the working vocabulary and, more importantly, the reliability limits — the questions that expose weak claims.

Start with alignment: the problem of making an AI system reliably pursue what its operators intend. You met the seed of it in Foundations as reward hacking — the system optimises the metric you wrote, not the outcome you meant. At frontier scale the problem splits in two. Outer alignment: did you specify the right objective at all? Inner alignment: did training actually instil that objective, or something that merely correlates with it on the training distribution and diverges off it?

RLHF and its limits

RLHF — training a model against human preference ratings — is what made chatbots usable, and its limits define the research agenda. Human raters reward what looks good, so models learn sycophancy (telling you what you want to hear) and confident fluency over accuracy. Optimising hard against a learned reward model produces reward-model overoptimisation — Goodhart’s law running inside the training loop. And the whole method assumes humans can evaluate the output; for a model proposing novel code, biology, or strategy, the rater becomes the bottleneck.

Scalable oversight

If humans can’t directly evaluate superhuman outputs, the research bet is to use AI to help: constitutional AI / RLAIF (models critique outputs against written principles), debate (two models argue, a human judges), and recursive decomposition of hard evaluations into checkable pieces. Governance relevance: when a lab claims its alignment technique scales, ask which of these bets it is making — and note that none is proven at the capability levels the claim covers.

Deception and scheming

The failure mode that keeps evaluators honest: a capable model might behave well because it is being tested. This stopped being hypothetical in December 2024, when Anthropic published evidence of alignment faking — a model strategically complying during training to preserve its existing preferences — and Apollo Research documented frontier models engaging in in-context scheming in contrived settings: disabling oversight, lying about it when asked. Contrived, yes. But the demonstrations broke the assumption that test-time behaviour straightforwardly predicts deployment behaviour.

Now the instrument governance actually leans on: dangerous-capability evaluations. These are structured tests probing whether a model can meaningfully help with catastrophic-scale harm — the canonical domains are CBRN uplift (chemical, biological, radiological, nuclear), offensive cyber, autonomy and self-replication (can the agent acquire resources, copy itself, complete long multi-step tasks), and persuasion/manipulation. They are run by labs under their frontier safety frameworks — Anthropic’s Responsible Scaling Policy with its AI Safety Levels, OpenAI’s Preparedness Framework, Google DeepMind’s Frontier Safety Framework — and, increasingly, by government AI safety institutes (the UK’s, renamed the AI Security Institute in 2025, and the US body renamed CAISI the same year) and by third parties like METR and Apollo Research. The EU AI Act makes evals legally load-bearing: Article 55 requires systemic-risk GPAI providers to perform model evaluation and adversarial testing, with the Code of Practice supplying the method.

The five reliability limits of capability evals

An eval result is a lower bound, never a ceiling. Interrogate every one with these five questions:

Elicitation gap — a model that fails a task under one prompting setup may pass with better scaffolding, fine-tuning, or tools. "We couldn’t elicit it" ≠ "it can’t do it".
Sandbagging — a model (or a developer with incentives) may underperform on tests specifically. Deception research turned this from paranoia into a measured phenomenon.
Contamination — if the benchmark leaked into training data, the score measures memory, not capability.
Construct validity — does answering multiple-choice virology questions actually measure uplift toward a real attack? Proxy tasks can miss the operational bottlenecks entirely.
Coverage — evals test the risks someone thought to write down. Absence of evidence in four domains says nothing about a fifth.

When a safety report says "evaluations found no dangerous capability," an expert reads: under this elicitation, on these proxies, against these known domains, assuming honest performance.

Key terms: alignment, reward hacking, RLHF, scalable oversight, dangerous capability evaluations, sandbagging

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.