Recovery without reinfection

Lesson 4 of 5 in Responding to Security Events.

Recovery is the phase with the highest pressure and the worst instincts. The business wants the service back, the responder wants to stop working at 03:00, and the fastest path — clean the compromised host in place and put it back in the load balancer — is the one that gets you a second incident next week.

The governing principle is rebuild rather than repair. You cannot prove a compromised operating system is clean; you can only prove it is less obviously dirty. An attacker with root may have replaced binaries, loaded a module, or hidden persistence in a place your check did not look, and the cost of being wrong is the entire investigation repeating with a worse timeline. So you build a new instance from known-good material — a hardened AMI from your pipeline, deployed by your infrastructure-as-code, with the vulnerability patched — and you restore data onto it from a point in time that predates the compromise.

Which puts the weight on one question: what time was that?

Recovery — and the gate that stops you returning to service too early

Containment holding, eradication performed
The attacker is stopped and persistence has been removed. Recovery is now allowed to begin — not before.
Is root cause identified and the entry point closed?
The honest check. If you do not know how they got in, rebuilding puts a fresh host behind the same open door. This gate is why recovery and root-cause analysis overlap in time rather than running in sequence.
Hold — keep it contained and finish the analysis
Staying down is a decision you can defend. Returning a vulnerable service to production because the clock was loud is not.
Rebuild from known-good material
A hardened AMI or container image from the build pipeline, deployed by infrastructure as code, with the vulnerability patched and the misconfiguration fixed. New resource, new credentials, new instance profile.
Establish the clean restore point
From the investigation timeline: the earliest confirmed attacker activity, minus a safety margin. Choose the last recovery point before that moment — and verify the backup itself has not been tampered with.
Restore data from before the compromise
AWS Backup recovery point, snapshot, or database point-in-time restore. Restore into the new resource, never over the compromised one, so the evidence survives the recovery.
Rotate every credential the attacker could have seen
Not only what was used: anything readable from the compromised position. Secrets Manager secrets the role could fetch, database passwords, API keys in environment or configuration, TLS private keys on disk, and any long-term key that existed at all.
Verify before serving traffic
Scan with Amazon Inspector, compare configuration against the baseline, confirm the investigation’s indicators appear nowhere, and re-run the detection that caught the original compromise against the new resource.
Verification clean?
A failed verification sends you back to the rebuild with new information — usually that the baseline itself was compromised.
Return to service with heightened monitoring
Phase traffic in, keep alerting tighter than normal for a defined watch period, and add detections for the specific indicators from this incident. Attackers return to environments that worked once.

Establishing the clean restore point is investigation work, not a storage decision, and it is the step that most often goes wrong. You need the earliest confirmed attacker activity, which is rarely the moment the detector fired — the finding at 14:07 may describe an access key created at 02:30 from a credential stolen three days earlier. The timeline comes from CloudTrail (the first unexpected call by the principal), Config (the first unexplained configuration change), the host’s own logs inside the disk snapshot, and the relationship view in Detective that shows when behaviour first departed from the baseline. Take that earliest timestamp, subtract a safety margin, and choose the last recovery point before it.

Then verify the backup itself. A restore point inside the compromise window is a reinfection vector, and a sophisticated attacker who had time will have gone looking for the backups — which is exactly why immutability and a separate account matter more than backup frequency. Restore to an isolated environment first where the stakes allow it, scan it, and only then promote it.

Where the replacement comes from — and how you establish that you can trust it
Restore source	When it is right	The risk	How you establish trust
Hardened AMI or image from the pipeline	Always, for the compute itself	The pipeline or the base image is itself compromised	Signed, versioned builds; scan the image with Inspector; confirm the build ran before the incident window
Infrastructure as code redeploy	Rebuilding the surrounding resources — security groups, roles, endpoints	The template repository was modified by the attacker	Review the commit history for the window; redeploy from a reviewed tag, not from `main`
EBS snapshot predating compromise	Restoring a volume’s data to a known state	The chosen snapshot is inside the compromise window	Pick it from the investigation timeline, not by recency; scan the restored volume before use
AWS Backup recovery point	Coordinated, cross-service restores with retention you can prove	Backups in the same account the attacker controlled	Vault Lock in compliance mode, cross-account copies, and a restore-testing history that shows it works
Database point-in-time restore	Transactional data where you need a specific second	The recovery window may not extend back far enough	Confirm the retention window covers the timeline before you rely on it
Cleaning the compromised host in place	Almost never — only when rebuild is genuinely impossible and the risk is accepted in writing	You cannot prove a root-compromised system is clean	There is no satisfying answer, which is the point: this row is the one the exam wants you to reject

Two more properties make recovery stick. Credential rotation is scoped by what the attacker could reach, not by what they used. If the compromised instance’s role could call secretsmanager:GetSecretValue on five secrets, treat all five as exposed — you have no way to prove they were not fetched unless data-level logging says otherwise, and "we saw no evidence of it" is not the same as evidence of absence. The same logic covers database credentials the application held, API keys in environment variables or parameter store, and private keys that sat on the disk.

And recovery is not complete until root cause is addressed. The service being up again is the business milestone; the incident closes when the entry point is shut, the persistence is gone, the detection gap that delayed you is fixed, and the control that would have prevented it is scheduled or deployed. That last item is what turns the loop back into the preparation phase, and it is the reason the lifecycle is drawn as a circle.

AWS Backup: the properties that matter during an incident

Backup plans schedule the work; backup vaults hold the recovery points; Vault Lock makes retention enforceable. Know the two modes: governance mode can be altered by a principal with sufficient permissions, while compliance mode, once locked after its cooling-off period, cannot be disabled or shortened by anyone — including the account root. That is the difference between a backup policy and a guarantee. Add cross-account and cross-Region copies so the recovery points do not live only where the attacker was, and restore testing, which periodically performs real restores and measures how long they took — converting "we have backups" into "we restored, last month, in 19 minutes". A logically air-gapped vault goes further, holding immutable copies that can be shared with another account for recovery without first copying them back.

The ransomware case, start to finish

Ransomware compresses every lesson in this module. Contain by isolating the affected hosts and revoking the credentials the encryption process was using — the identity is usually the spread mechanism, not the network. Capture before anything else: the memory may hold the key material, and the encrypted volumes are evidence. Assess which data stores are affected and whether backups were targeted, which is the question that decides whether this is an outage or a crisis. Recover from immutable backups in a separate account, choosing a recovery point before the first encryption event. Do not pay on the assumption the data comes back; your recovery plan is the actual mitigation. Then look at prevention honestly: Vault Lock in compliance mode, cross-account copies, versioning with MFA delete on critical buckets, least privilege on the delete and encrypt APIs, and S3 Object Lock where immutability is a requirement rather than a preference.

Why the restore point comes from the investigation, not from the backup console

The detector’s timestamp is when the pattern matched, which may be days after initial access. Build the timeline from the first unexpected CloudTrail call by the principal, the first unexplained AWS Config change, the earliest suspicious entry in the host logs inside your disk snapshot, and the point at which Detective shows behaviour departing from the baseline. Then pick the last recovery point before the earliest of those, with margin. Choosing by recency is how teams restore the compromise along with the data.

What "verified before returning to service" means concretely

Five checks, all cheap. Scan the new resource with Amazon Inspector for the vulnerability class that let them in. Diff its configuration against the known-good baseline — AWS Config or your IaC plan will do it. Confirm none of the investigation’s indicators (file hashes, domains, IPs, IAM principal names) appear on or around the new resource. Re-run the detection that caught the original compromise and confirm it is now armed and quiet rather than quiet because it is broken. And phase traffic in with tighter alerting than normal for a defined watch period, because the single best predictor of being attacked is having been attacked.

Key terms: Known-good material, Eradication, Containment

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.