Red-teaming and the adversarial threat catalogue

Lesson 5 of 6 in Testing and Evaluation: Fairness Math, Red-Teaming, and Explainability.

Everything so far tests how the system behaves for honest users. Red-teaming tests how it behaves for adversaries and for the worst honest user on the worst day — deliberate, structured attempts to make the system fail, before someone outside does it for you.

Run as a governance function (not an ad-hoc hackathon), a red-team engagement has the same skeleton as a penetration test: a scope (which harms are in bounds — jailbreaks? data leakage? bias elicitation? dangerous-capability uplift?), rules of engagement (real user accounts or sandboxes, what may be exfiltrated as proof, who is notified), coverage tracking against a threat taxonomy so you can say what was not tested, and a findings pipeline that routes results into severity grading and release gates. Manual expert probing finds novel failure classes; automated red-teaming (attack libraries, adversarial-prompt generators, model-vs-model attacks) provides the regression coverage that keeps old holes closed on every new version. Mature programs run both, plus periodic external red teams for independence — the same logic as financial audit.

Regulation has caught up: the EU AI Act requires adversarial testing of systemic-risk GPAI models (Art 55), the GPAI Code of Practice operationalises it, and US frontier-lab safety frameworks (Anthropic’s RSP, OpenAI’s Preparedness Framework) build their capability thresholds on red-team evidence. The technique graduated from craft to compliance artefact.

To scope a red team you need the threat vocabulary. MITRE ATLAS — the adversarial-ML sibling of ATT&CK — catalogues the tactics; the OWASP Top 10 for LLM applications does the same for the genAI application layer, with prompt injection at number one. The core attack classes every governance professional must distinguish:

• Evasion — craft inputs the live model misclassifies (adversarial examples: the sticker that makes a stop sign read as speed limit).
• Data poisoning — corrupt the training data to plant backdoors or degrade performance.
• Model extraction — reconstruct a copy of the model by querying it at scale (IP theft plus a staging ground for other attacks).
• Model inversion & membership inference — recover training data, or confirm a specific person’s record was in the training set (a privacy breach with GDPR consequences).
• Prompt injection — smuggle instructions into an LLM’s input, directly or via content it retrieves (the indirect variant), hijacking tools and data the model can reach.

Interactive sorting exercise: Name the attack class in each incident report.

Testing only governs if its results can stop a launch. That is the release gate: acceptance criteria fixed before results exist (accuracy floor, per-group performance bands, impact-ratio limits, red-team severity ceilings), a named sign-off owner, and the rule that model updates re-run the full gate — regression evals catch the fix that silently broke something else. Champion/challenger designs extend the gate into production: the incumbent model keeps serving while a challenger must beat it on the gated metrics before promotion.

Key terms: red-teaming, prompt injection, data poisoning, membership inference, release gate, mitre atlas

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.