What Red Teaming Is
Lesson 1 of 3 in Red Teaming LLMs.
Red teaming is adversarial probing of your own system: you attack the model on purpose, before adversaries and ordinary users do it by accident. The name comes from military exercises where a “red team” plays the enemy against the defending “blue team”, and the LLM version keeps the essential property — the attacker adapts. A red teamer reads the model’s refusals, reframes, escalates across turns, and combines techniques until something gives. That is a different activity from running an eval, and the difference is worth being precise about.
A Benchmark is static: a fixed, usually public question set, identical for every model, scored once. Red teaming is adaptive: the next probe depends on the last response. A benchmark measures average performance on known cases; a red team hunts worst-case behavior on cases nobody wrote down yet — it is coverage-seeking, deliberately spreading probes across attack techniques and harm categories rather than sampling one distribution. Penetration testing, the third sibling, shares the adversarial mindset but aims at a different layer: infrastructure, credentials, and code around the model rather than the model’s behavior itself.
This is a published discipline, not folklore. Ganguli et al. (2022) documented manual red teaming at scale — recruited red teamers attacking assistant models turn by turn, with the resulting dataset of attack transcripts released for study — and reported, among other things, that models trained with Reinforcement learning from human feedback (RLHF) became harder to red team as they scaled, while the model configurations without RLHF training showed no such trend. Perez et al. (2022) automated the practice: language models generating attacks against other language models, with a classifier flagging the failures. You will meet both methods properly in the next lesson.
| Benchmarking | Red teaming | Penetration testing | |
|---|---|---|---|
Question it answers | How well does the model do on known, representative tasks? | What can a motivated adversary make the model do? | Can an attacker breach the system around the model — network, credentials, code? |
Test material | Fixed question set, usually public, same for every model | Adaptive probes that change based on the model’s responses | Exploits against infrastructure, APIs, and configuration |
Adversary model | None — the test does not fight back | Explicit: human or automated attacker, adapting | Explicit: attacker with technical access goals |
Output | A score to compare models | Findings: reproducible transcripts of failures, triaged into fixes | Vulnerabilities with severity ratings and remediations |
Failure it protects against | Shipping a model worse than the last one | Shipping behavior you never thought to test | Shipping an exposed system |
What a model-level red team probes. The scope of this module is the model’s own behavior: jailbreaks that defeat refusals, Prompt injection handling, training-data leakage and System prompt leakage, harmful or disallowed content, and Hallucination under pressure — the confident confabulation that appears exactly when a user pushes into territory the model should decline or hedge. Each of these is a behavior of the token predictor itself, which is why the earlier modules in this domain — the single-channel problem, the jailbreak taxonomy, the memorization results — are the red teamer’s working knowledge.
Two neighboring scopes get one line each. Red teaming an agent — where the model holds tools and the kill chain runs through them — adds an attack surface this site deliberately does not teach; our sister AI Agent Academy owns it. And the formal obligation to red team — who must do it, when, with what documentation — is governance, covered by the AI Governance Academy. Here we care about the craft: how to find model-level failures, and what to do with them.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.