Training and the moment of truth: evaluation

Lesson 3 of 5 in The AI Lifecycle: From Data to Deployment to Retirement.

Stage 3 is where data becomes behaviour. For most systems this is a matter of hours on rented cloud GPUs. For frontier models it is something else entirely: months of training across tens of thousands of chips, at costs estimated in the hundreds of millions of dollars — which is why only a handful of companies can afford to play, why compute became a regulatory proxy (the EU’s 10²⁵ FLOPs threshold), and why the AI supply chain concentrates so much power upstream.

Training itself has governance-relevant layers, especially for generative models:

Pre-training

The expensive foundation. The model learns general patterns from an enormous corpus — for LLMs, trillions of tokens of text. Almost every controversial data question (scraping, copyright, personal data, CSAM contamination) lives here, because this is where the vast undifferentiated data goes in. Once pre-training ends, whatever the model absorbed is effectively baked in.

Fine-tuning

Specialisation. A smaller, curated dataset adapts the general model to a task or domain — legal drafting, medical coding, your company’s tone. This is the layer where most downstream actors touch training, and where responsibility starts to shift along the value chain: your fine-tuning data, your choices, your accountability.

Alignment

Teaching preferences. Techniques like RLHF tune the model toward outputs humans rate as helpful and harmless — this is what turned raw text predictors into usable assistants. Governance cares because alignment choices encode values (whose raters? which refusals?) and because alignment is imperfect: jailbreaks are, precisely, ways of routing around it.

Along the way, engineering hygiene doubles as governance evidence: checkpoints and versioning mean you can say exactly which model, trained on which data, made which decision — the traceability that incident investigation and legal accountability both depend on. A team that cannot reproduce its own model cannot answer a regulator’s first question.

Then comes stage 4, the moment of truth: does the thing actually work — and for whom? Four kinds of scrutiny have become standard, and each produces a documentation artifact that later stages (and regulators) rely on.

Benchmarks — and why acing them proves less than it seems

Standardised test sets let you compare models on a common yardstick. But benchmarks are narrow, public (so models can effectively memorise them through training-data contamination), and often unlike your real deployment context. A model that scores 92% on a benchmark can still fail your population. Governance rule of thumb: benchmark scores are the beginning of evidence, never the end.

Disaggregated testing and bias audits

Aggregate accuracy hides subgroup failure — the central lesson of Gender Shades, which found commercial face analysis was near-perfect on lighter-skinned men and failed on up to a third of darker-skinned women. Testing must be broken out by relevant subgroups, which is exactly what NYC Local Law 144 mandates for hiring tools: an annual independent bias audit with published selection rates by race and sex.

Red-teaming and safety evaluations

Structured adversarial attack: experts (and increasingly, automated attackers) try to make the system misbehave — jailbreaks, prompt injection, dangerous-capability probes. Frontier-model rules have made this quasi-mandatory: the EU AI Act requires adversarial testing for GPAI models with systemic risk, and safety institutes like the UK AISI run pre-release evaluations of frontier models.

Model cards and system documentation

A model card records what the model is, what data trained it, how it performs across subgroups, and what it should not be used for. Proposed by Margaret Mitchell, Timnit Gebru and colleagues in 2019, the idea graduated from research paper to industry norm to legal expectation — the EU AI Act’s technical-documentation duties are, in essence, model cards with legal force.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.