Patching at scale
Lesson 4 of 6 in Securing Compute Workloads.
Patching a fleet by hand does not scale past the first incident. Systems Manager Patch Manager turns it into four objects you configure once and then mostly leave alone.
A patch baseline decides what counts as an approved patch. It holds approval rules — by product, classification (security updates, bug fixes, critical updates) and severity — each with an auto-approval delay in days, or an approve-until date. The delay is the quiet professional detail: "approve security updates seven days after release" means a vendor's bad patch has a week to be withdrawn before your fleet installs it, while still giving you an upper bound on exposure. Baselines also carry explicit approved and rejected patch lists for the one package you must pin, and a compliance severity so a missing patch is reported at the level you care about. AWS supplies default baselines per operating system; production environments usually run a custom one.
A patch group decides which instances a baseline applies to: tag instances with the Patch Group tag key and register that value against a baseline. This is how dev gets patches on day one and production gets them on day eight, from the same approval rules.
A maintenance window decides when. A schedule, a duration, a cutoff after which no new task starts, registered targets, and a registered task — normally the AWS-RunPatchBaseline document — with concurrency and error thresholds so a bad patch stops after five instances instead of five hundred.
And the operation decides whether anything changes: Scan evaluates compliance and installs nothing; Install applies approved patches and reboots if the patch requires it (or defers the reboot if you tell it to). Most organizations scan continuously and install on a window.
One patch cycle, from tag to compliance record
- Instances tagged Patch Group = prod
The tag is the join key between a fleet and a baseline. Untagged instances fall to the default baseline for their OS — which is rarely what you intended.
- Baseline approval rules evaluate each patch
By product, classification and severity, with an auto-approval delay in days, plus explicit approve and reject lists.
- Has the auto-approval delay elapsed?
A seven-day delay lets a defective vendor patch be withdrawn before your fleet takes it, while bounding how long you stay unpatched.
- Patch stays unapproved this cycle
It will be approved on a later run — and in the meantime the instance is reported as compliant against this baseline, which is why baseline design is a risk decision.
- Maintenance window opens
Schedule, duration, cutoff, concurrency and error thresholds. The thresholds are the blast-radius control: stop after N failures.
- Scan or Install?
AWS-RunPatchBaseline takes an Operation parameter. Scan is read-only; Install changes the instance and may reboot it.
- Scan: report only, change nothing
Run this often — daily, via State Manager — so compliance data is never stale between windows.
- Install approved patches, reboot if required
Reboot behaviour is a parameter; NoReboot leaves the patch installed but possibly inactive, which compliance will keep reporting.
- Compliance state published
Visible in Patch Manager and Systems Manager Compliance, reported into Security Hub, and checkable by an AWS Config rule so drift becomes a finding rather than a surprise.
| Patch in place (Patch Manager) | Rebuild the image (immutable) | |
|---|---|---|
How a CVE gets fixed | approved patch installed on the running instance in a maintenance window | new AMI or container image built from a patched base, then rolled out by replacing instances or tasks |
Fits | long-lived instances, stateful servers, licensed software, anything you cannot casually replace | stateless fleets behind a load balancer, containers, anything already built by pipeline |
Rollback | uninstall or restore from snapshot — slow and sometimes messy | point the launch template or task definition back at the previous image version |
Drift | accumulates — two instances with the same tag can diverge over a year | impossible by construction: every instance is a fresh copy of a tested image |
Evidence for an auditor | patch compliance reports per instance, per baseline | the image version an instance booted, plus the pipeline’s test results for that version |
The cost | reboots, maintenance windows, and the drift you inherit | build pipeline engineering, and workloads must tolerate replacement |
Continuous validation is the half of patch management teams forget — and the exam names it in the same breath as automating the updates themselves. Automating the update is not the control; proving it happened is. Three mechanisms stack up: run Scan on a daily State Manager association so compliance data is fresh between windows; let Patch Manager publish non-compliance into Security Hub so patch state sits beside your other findings; and add an AWS Config rule for managed-instance patch compliance so a drifted instance becomes a tracked, remediable finding with a timeline. Amazon Inspector closes the loop from the other direction — it keeps scoring the CVEs that are still present regardless of what your patch baseline approved, which is exactly the independent check an auditor wants.
At organization scale, patch policies in Systems Manager Quick Setup configure baselines, scanning and installation schedules across accounts and Regions from one place, instead of per-account maintenance windows that slowly diverge. The same reasoning as Firewall Manager at the edge: consistency is itself the control.
One honest caveat on both strategies. Patching reduces known vulnerability exposure; it does not make a compromised instance trustworthy. If GuardDuty says a host is executing an attacker's binary, the answer is isolation and rebuild — not Install.
Key terms: Patch baseline, Patch group, Maintenance window, Immutable infrastructure
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.