Running a Round

Lesson 3 of 3 in Red Teaming LLMs.

A red-team round starts with scoping, and scoping is two decisions. Which behaviors: pick the harm categories that matter for this application — an internal coding assistant and a public health chatbot do not share a worst case — and be explicit about what is out of scope this round. Which threat model: who is the adversary you are simulating? A curious user typing into the chat box, a hostile author of content your RAG pipeline might retrieve, a competitor scripting your API? The threat model decides which technique families from the last lesson you run and which grid cells you must fill. Pin down the exact system under test — model identifier and version, System prompt, Content filter configuration, sampling settings — because a finding against an unpinned system is a rumor, not a result.

One red-team round, scope to retest

  1. Scope

    Behaviors in and out of scope, threat model, technique families to run, and the pinned system under test: model version, prompt, filters, sampling settings.

  2. Probe

    Run the chosen methods — manual, automated, suffix search, suite replay — against the pinned configuration at production settings.

  3. Log & reproduce

    Every candidate finding gets a full transcript plus the configuration that produced it, and is reproduced before it is filed. Sampling is stochastic — record the settings and the hit rate, not just one lucky transcript.

  4. Triage each finding

    Severity × reproducibility × fixability. Every finding leaves triage on exactly one of four routes.

  5. Fix in prompt

    Tighten instructions, restructure context, add explicit refusal guidance. Cheap and fast — but a disposition, not enforcement.

  6. Fix in filters

    Input/output classification outside the model: content filters, PII masking, pattern blocks. Enforcement the model cannot be talked out of — with false-positive costs.

  7. Accept with monitoring

    Low severity, low frequency, or a fix that would cost more than the harm: document the decision, set a production monitor, and revisit on a schedule.

  8. Escalate

    Model-level failures no app layer can fix (e.g. memorized-PII emission), or severity that blocks launch: raise to security leadership and report provider-side issues through the provider’s disclosure channel.

  9. Regression-test the fix

    The finding becomes a permanent test. Verify the fix, check for collateral damage on your quality evals, and rerun the test on every future model or prompt change.

  10. Round report — the suite grows

    Coverage grid with probe and failure counts, triage decisions with owners, and a regression suite larger than the round started with.

Execution discipline is what makes a round worth its cost. Log everything: exact prompts and responses, model identifier and version, sampling parameters, filter configuration, timestamps. Outputs are stochastic under Sampling, so a finding is not “this prompt broke the model once” but “this prompt breaks the model at roughly this rate under these settings” — reproduce before you file. Deduplicate aggressively; a hundred paraphrases of one failure are one finding. And keep the attack transcripts access-controlled: a red-team log is a cookbook of working attacks against your own system, which makes it one of the most sensitive artifacts your team produces.

Triage sends every confirmed finding down exactly one route, and the routes have different physics. A prompt fix changes the model’s instructions — cheap, immediate, and only as strong as the model’s trained disposition to obey; the injection module explains why that disposition can be defeated. A filter fix adds enforcement outside the model, which holds against persuasion but pays a false-positive tax. Accept with monitoring is a legitimate engineering decision for low-severity findings — made explicitly, documented, with a production monitor watching the rate. Escalate is for what you cannot fix: model-level behavior like memorized-data emission belongs with the model provider, and launch-blocking severity belongs with leadership. Whatever the route, the finding’s final form is the same: a regression test, rerun on every model and prompt change — the machinery of Production Evals and Regression Testing. Whether a regulator obliges you to run such rounds, and what documentation they expect of them, is the AI Governance Academy’s territory; here the duty is engineering: fixes without retests are wishes.

Interactive sorting exercise: Triage these red-team findings. Send each one down the route a disciplined round would choose.

In production

Red teaming is not a launch ceremony — it is a recurring gate, like load testing: before launch, and again after every major model or prompt change. That cadence is the practice all three clouds’ safety documentation and the OWASP LLM Top 10 guidance assume, because every documented safety surface is configurable — and a configuration you changed is a configuration you have not tested.

AWS

Amazon Bedrock Guardrails is built for exactly this loop: guardrails are versioned with a working draft and a built-in test window, so a red-team round can probe a draft configuration before it is promoted, and the ApplyGuardrail API evaluates content independently of model invocation — letting your harness test filter behavior (content categories including Prompt Attack, denied topics, sensitive-information filters) in isolation from the model. Rerun the round when either side changes: the guardrail version or the model behind it.

Azure

Azure AI Foundry attaches content-filter configurations (powered by Azure AI Content Safety, with optional Prompt Shields for prompt attacks and indirect prompt injection) to individual model deployments — so a new deployment or a modified filter configuration is a new system under test. Outcomes are machine-readable for a probing harness: blocked prompts return an HTTP 400 error and filtered completions signal a content_filter finish_reason, which lets your regression suite assert on filter hits versus model refusals automatically.

Google Cloud

Vertex-style serving reports why content was blocked through explicit enums (block and finish reasons such as SAFETY, RECITATION, or SPII), so a harness can distinguish a model refusal from a filter interception — two different defense layers that a round must probe separately. Configurable content filters score four harm categories against adjustable thresholds while non-configurable filters block prohibited content outright; the docs point to system instructions for steering the model itself, since filters act as a barrier rather than changing the model. Threshold changes, instruction changes, and model updates each warrant a retest.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.