The Methods
Lesson 2 of 3 in Red Teaming LLMs.
No single method covers the attack space, so mature red-team programs layer several. Five families do most of the work in the published literature: manual expert probing, crowdsourced probing, automated model-vs-model attack generation, optimization-based suffix search, and curated adversarial suites that replay known attacks. Each finds a different kind of failure, and each has a blind spot the next one exists to cover.
| Method | How it works | What it finds best | Blind spot |
|---|---|---|---|
Manual expert probing | Security- and domain-experts attack the model turn by turn, adapting to each response | Novel techniques; multi-turn social framings; judgment calls on borderline harm | Expensive and slow — coverage bounded by hours and imagination |
Crowdsourced probing | Many non-expert red teamers attack in parallel — the setting Ganguli et al. (2022) documented at scale, releasing the attack transcripts | Diversity: framings, topics, and phrasings no small expert team would produce | Shallow depth per attacker; results need classification and human audit |
Automated model-vs-model | An attacker LM generates test prompts; a classifier flags the target’s failures (Perez et al. 2022) | Scale — hundreds of thousands of probes, exposing clusters of failures | Only finds what the classifier can recognize; attacker diversity must be managed |
Optimized adversarial suffixes | Gradient-guided search (GCG, Zou et al. 2023) builds an Adversarial suffix that pushes the model past refusal — suffixes transfer across models | Failures in the space of token sequences no human would ever type | Needs gradient access to optimize (transfer softens this); odd-looking strings that naive input filters may catch — an arms race, not immunity |
Curated adversarial suites | Replay published harm-behavior prompt sets — such as the behavior suite released alongside the GCG paper — plus your own past findings | Regression: cheap, repeatable, comparable across model versions | Static — public suites saturate and can leak into training, inheriting every benchmark weakness |
Key terms: Red teaming, Jailbreak, Adversarial suffix, Content filter, Refusal
The automated loop deserves a closer look, because it changed the economics of the whole practice. Perez et al. (2022) used one language model to attack another: the attacker generates candidate test prompts, the target responds, and a classifier trained to recognize the failure (offensive content, in their headline experiments) scores every reply. They escalated the attacker through zero-shot generation, few-shot prompting seeded with past successes, supervised fine-tuning on successful attacks, and reinforcement learning against the classifier’s signal — each stage trading some diversity for a higher hit rate — and surfaced tens of thousands of offensive replies from a 280-billion-parameter chatbot along the way. The design generalizes to any failure you can build a detector for: leakage, System prompt leakage, Personally identifiable information (PII) emission, harmful instructions.
The GCG result belongs in the same toolbox with a different engine: instead of a model writing attacks, an optimizer searches for them, token by token, guided by gradients toward outputs that begin with compliance instead of Refusal. The full mechanism — and why the suffixes transfer to models the attacker never had weights for — is in Prompt Injection and Jailbreaks; what matters here is what it contributes to a red team: probes from a region of input space that manual and model-generated attacks never reach.
An automated red-team loop (after Perez et al. 2022)
- Define the target failure
Pick one behavior to hunt — a disallowed-content category, system-prompt leakage, PII emission — and write down what counts as a hit. This definition becomes the classifier’s job description.
- Attacker model generates probes
Zero-shot from an instruction, few-shot seeded with past successes, or fine-tuned/RL-trained on the classifier’s signal — progressively higher hit rates, progressively narrower diversity.
- Target model responds
Run at production settings and log everything: exact prompt, response, model identifier and version, sampling parameters.
- Classifier scores every response
Flags failures at a scale no human review could match. Audit samples by hand — the classifier errs in both directions, and its misses are your blind spot.
- New failure modes still appearing?
Cluster the hits. If new clusters keep forming, the space is not exhausted; if the same failures repeat, this loop has given what it can.
- Seed attacker with successes
Successful attacks become few-shot examples or training data for the attacker — the loop sharpens itself.
- Findings pool → triage
Deduplicated, reproducible failures move to the triage stage of the next lesson.
Coverage thinking: the technique × harm grid
The honest unit of red-team reporting is not “we attacked it for a week” — it is a grid: attack-technique families down one axis, harm categories across the other, and in each cell the number of probes you actually ran and the failures you actually found. The grid makes two things impossible to hide. First, empty cells are claims you cannot make: if nobody probed leakage with optimized suffixes, you know nothing about leakage under optimized suffixes. Second, methods concentrate in predictable cells: manual rounds gravitate toward role-play and escalation framings because those are what humans invent; automated loops concentrate where the classifier sees clearly; replayed suites only cover last year’s attacks.
| Technique family ↓ / Harm → | Disallowed content | Training-data / PII leakage | System-prompt leakage | Confident misinformation |
|---|---|---|---|---|
Role-play & persona framings | Manual + crowdsourced probe this heavily | Rarely tried by hand — a classic gap | Manual rounds reach it | Manual rounds reach it |
Encoding & obfuscation tricks | Suites replay known ones; new ones need experts | Seldom probed at all | Occasionally probed | Almost never probed — gap |
Multi-turn escalation | Manual’s home turf | Needs patient manual work | Manual’s home turf | Manual reaches it; automation struggles with long horizons |
Optimized suffixes (GCG-style) | The published GCG target | Possible where a detector exists | Possible where a detector exists | Hard — no crisp optimization target |
Indirect injection via retrieved content | Needs a RAG-shaped harness to probe at all | Needs a harness — gap in most programs | Needs a harness | Needs a harness |
This is why single-technique testing under-covers systematically, not just statistically: every method has cells it structurally cannot reach. An all-manual program never samples the suffix-search region; an all-automated program is blind wherever its classifiers are; a replay-only program is a Benchmark wearing a red shirt. Budget rounds by filling the emptiest important cells first, and treat a cell’s emptiness — not a low failure count — as the finding to report upward.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.