Testing the plan before you need it

Lesson 3 of 3 in Automated Response and Forensics.

An untested incident response plan is a document about an incident response plan. The exam knows this, which is why "how would you validate the effectiveness of the IR plan?" is a question with a concrete, service-level answer rather than a sentiment about culture.

The answer has three layers, and they test different things. A game day tests the system: inject a real fault, let the real detectors fire, let the real responders respond, and watch what breaks. A tabletop exercise tests the people and the decisions: walk a scenario verbally around a table, and discover that nobody knows who authorises disabling a production role or who tells the regulator. And automated checks test the preparation itself, continuously: is the responder role still deployed in every account, is the SSM Agent still reporting, does a sample finding still reach the remediation rule? Each layer finds failures the others cannot. A tabletop will never find a StackSet that missed an account; an automated check will never find that two teams believe they each own containment authority.

The three validation layers — what each one actually proves
	Tabletop exercise	Game day	Automated check
Tests	Decisions, roles, communications	The whole socio-technical system under a real fault	That the preparation still exists and still works
Costs	An hour of several people’s time	A planned window, real blast radius, real risk	Almost nothing, once built
Typical finding	"Nobody agreed who notifies the regulator, or within how long"	"The runbook’s snapshot step fails because an SCP denies it in that OU"	"The IR role is missing from the three accounts created in September"
AWS tooling	None — it is a conversation against a written playbook	AWS Fault Injection Service, plus GuardDuty sample findings to exercise the security path	AWS Config rules, scheduled Automation documents, Security Hub controls, a canary finding on a schedule
Frequency	Quarterly, and after any major change	A few times a year per critical workload	Continuous
Blind spot	Everything technical — people describe what they believe happens	Expensive, so it covers few scenarios	Only checks what you thought to check

AWS Fault Injection Service is the named tool, and its model is small enough to learn in a paragraph. You write an experiment template containing actions (from a library — stop or terminate instances, send an SSM command, disrupt network connectivity, inject API errors, interrupt power to an Availability Zone), targets (resources selected by tag, id, or filter, with selection modes like "all" or "count 2"), an IAM role FIS assumes to act, and — the part that makes it safe — stop conditions: CloudWatch alarms that, if they fire, halt the experiment and roll it back. FIS also ships a scenario library of pre-built experiments for common failure modes, so you do not have to compose an AZ power interruption from primitives.

Be precise about what FIS is for, because this is where candidates over-claim. FIS injects faults, not attackers. It will not create a GuardDuty finding or simulate credential exfiltration. For the security path you pair it with signals you can generate safely: GuardDuty sample findings, which produce realistic finding types on demand and are the standard way to prove that your finding → EventBridge → remediation chain still works end to end, and Security Hub’s own sample and custom findings for the aggregation path. Use FIS for "can we survive and recover from this disruption", and sample findings for "does our detection and response machinery still fire".

The game-day loop — and the two gates that produce the backlog

Write the hypothesis
A testable sentence: "If an instance starts mining cryptocurrency, GuardDuty raises a finding within 15 minutes, our automation isolates it within 5, and no production traffic is affected."
Define the experiment: actions, targets, stop conditions
FIS template with the action library, tag-based targets, and CloudWatch alarms as stop conditions. For security paths, add the GuardDuty sample finding that stands in for the attacker.
Run it — pre-production first, then production in a window
The first run belongs in a copy of production. A game day that takes down the real service teaches you one thing and costs you the next three.
Did detection fire, and inside the target time?
Measure mean time to detect from the injected fault to the finding or alarm. "It fired eventually" is a failed hypothesis.
Fix the detection gap
A missing log source, a suppressed finding type, an alarm with no action, a rule pattern that did not match. These are Content Domain 1 fixes surfaced by a Domain 2 exercise.
Did the runbook work exactly as written?
Not "did we recover" — did the documented steps work, with the documented permissions, for someone following them literally? The commonest failure is a step that needs a permission the responder role does not have.
Fix the runbook, or the permission it assumed
Edit the document, widen or narrow the IR role, remove the SCP that blocked the step. Then the change itself needs re-testing.
Record detection and response times
Time to detect, time to engage, time to contain, time to recover — tracked over successive game days so improvement is visible rather than asserted.
Feed findings into preparation; schedule the re-run
Every finding becomes a backlog item with an owner, and the experiment goes on a schedule so a regression is caught by the exercise rather than by an attacker.

Interactive sorting exercise: Each question below is best answered by one of the three validation layers. Sort them — the point is that no single layer can answer all of these.

Measurement is what turns exercises into an argument you can take to a budget meeting. Track four intervals and track them over time: time to detect (fault injected → finding or alarm raised), time to engage (finding raised → a human acknowledged), time to contain (engagement → the spread stopped), and time to recover (containment → service verified and back). Each belongs to a different fix: detection time is a logging and rule problem, engagement time is an on-call and escalation problem, containment time is a runbook and permissions problem, recovery time is an architecture and backup problem. A single "mean time to resolve" number hides which of the four is broken.

Then close the loop properly. Post-incident analysis and post-exercise review produce the same artefact: a list of findings, each with an owner and a due date, feeding straight back into the preparation phase — a runbook edit, a widened IR role, a new detection rule, an added exclusion tag, a Resilience Hub re-assessment. If the exercise produced no backlog items, it was a demonstration, not a test.

"How would you validate the IR plan?" — the answer in one breath

Run game days with AWS Fault Injection Service (experiment templates with actions, tag-based targets, and CloudWatch alarm stop conditions), generate GuardDuty sample findings to prove the detection-to-remediation chain still fires, hold tabletop exercises against the playbooks to test decisions and notifications, check preparation continuously (responder-role coverage, agent coverage, log delivery), measure detection and response times, and feed every finding back into the plan. Use Resilience Hub to generate the experiments and procedures you did not think to write.

And know the boundary: FIS injects faults, not attackers. A question asking how to simulate a GuardDuty-detectable compromise wants sample findings; a question asking how to verify recovery from infrastructure failure wants FIS.

Key terms: AWS Fault Injection Service (FIS), Game day, AWS Resilience Hub

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.