Triage: validating what you were told

Lesson 1 of 5 in Responding to Security Events.

SCS-C03 added a skill to this task that SCS-C02 did not have: validating findings from AWS security services to assess the scope and impact of an event. It is the most useful addition in the whole guide, because it names the discipline that separates a responder from a button-pusher.

Here is the mental model. A finding is a hypothesis. GuardDuty is telling you that something it observed resembles a pattern associated with compromise. It is a well-calibrated, expensively-built hypothesis, and it deserves your attention — but it is not a verdict. An EC2 instance performing DNS lookups against a domain on a threat list might be compromised, or might be a security researcher’s sandbox, or might be your own monitoring agent resolving a domain that got listed last week. A security incident is what you have once you have validated the hypothesis and established that something actually happened. Everything downstream — severity, engagement, containment authority, the regulatory clock — hangs off that transition, so a responder who skips validation either over-responds (isolating production because a detector had an opinion) or under-responds (closing a real compromise as noise because it looked familiar).

Validation is also where scope comes from, and scope is the thing containment needs. Isolating the one instance named in the finding is not a response if the credential that was stolen from it has been used in four other accounts.

Triage — the sequence that turns a finding into a scoped incident

  1. A finding arrives

    From GuardDuty, Security Hub, Inspector, Macie, a Config rule, a CloudWatch alarm, or a human report.

  2. Read it properly: type, severity, resource, principal, window

    The finding type tells you what pattern matched, which tells you what evidence would confirm or refute it. Note the exact time window — every subsequent query depends on it.

  3. Confirm the resource and the principal are what the finding says

    Does the instance still exist? Which account and OU? What is its role, its tags, its owner? Is the principal a human identity, a role assumed by a workload, or a long-term access key? Each answer points at a different playbook.

  4. Pull the CloudTrail events around the window

    The finding says a pattern matched; CloudTrail says what the principal actually called, from where, and with what result. This is where a hypothesis becomes evidence or dissolves.

  5. Is this behaviour expected for this principal?

    Two ways to get this wrong: calling real activity "expected" because it looks familiar, and calling expected activity an incident because it looks alarming. Verify with the resource owner rather than assuming either way.

  6. Not an incident — suppress precisely, and record why

    A suppression rule scoped to this finding type and this resource, with a written reason and a review date. A blanket suppression of the finding type is how the next real one gets missed.

  7. Classify severity against the published table

    Your severity, not the detector’s. A MEDIUM finding on a production database holding personal data outranks a HIGH finding in a sandbox. The finding’s severity is one input.

  8. Scope it: what else did this principal touch?

    Every API call by this principal in the window, every resource it reached, every other principal it assumed or created, and whether the same source IP appears elsewhere. This is what containment has to cover.

  9. A scoped incident you can act on

    Confirmed, classified, and with a blast radius written down — the handover containment actually needs.

Walk a real one. A GuardDuty finding of type UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS arrives at 14:07 UTC against role AppServerRole in the production account.

Read it. The type is specific and valuable: it says credentials belonging to an EC2 instance profile were used from an IP address outside AWS. That is a narrow, hard-to-explain-away pattern — far more specific than "unusual API call" — and it tells you exactly what to confirm.

Confirm the subjects. AppServerRole is the instance profile on the web tier. The finding names the instance; the instance exists, is running, and is internet-facing behind a load balancer. Now you know the hypothesis is coherent.

Pull CloudTrail. Filter on the role session in a window around 14:07 and look at sourceIPAddress. You find calls from the instance’s own address all morning, and then from 14:03, calls from an address in a hosting provider’s range: s3:ListBuckets, s3:GetObject against two buckets, iam:GetUser, iam:CreateAccessKey. That last call is the moment this stopped being a hypothesis.

Decide: expected? No. The web tier has never called iam:CreateAccessKey, and legitimate instance credentials do not originate outside AWS. Classify: confirmed credential compromise with attempted persistence in production — the top tier. Then scope, which is the step that decides whether your containment works: which buckets were read, which new access key was created, what has that key done since, and does the same source address appear against any other principal in any other account? That question is a graph traversal, and it is what Detective exists for.

Which principal — and why it changes everything

Three cases, three playbooks. A role assumed by a workload (an instance profile, a Lambda execution role) means temporary credentials, so you are looking for exfiltration and your containment lever is session revocation plus the resource that held them. A long-term IAM user access key means a static secret that may be in a repository, a laptop, or a CI system, so containment is deactivation and the investigation extends to wherever the key was stored. A federated human identity means the compromise may be in the identity provider, not in AWS at all — and containment includes the provider’s session, not just the AWS role. Read the userIdentity block in CloudTrail before choosing a playbook.

What CloudTrail tells you that the finding cannot

The finding describes a pattern; CloudTrail describes actions. For each call in the window you get the principal and session, the sourceIPAddress and userAgent, the exact API and parameters, the result (including errorCode — a run of AccessDenied is an attacker mapping their permissions), and the request id you can cite. Search it with CloudTrail Lake or Athena rather than by scrolling the console, and remember the boundary: management events are on by default, data events are not — if nobody enabled S3 data events, "which objects were read?" may be unanswerable, which is itself a finding for the post-incident backlog. The query craft lives in Log Analysis.

The two ways to get "is this expected?" wrong

Over-familiarity: the activity resembles something legitimate, so you close it. Attackers deliberately use normal-looking APIs from normal-looking agents; "our backup job does that" needs to be checked against the backup job, not remembered. Over-alarm: the activity looks dramatic, so you isolate production — and the dramatic thing was a penetration test, a new deployment, or an engineer debugging. The discipline is the same in both directions: get a verification, from the resource owner or the change record, and write it down. An unverifiable "expected" is treated as unexpected.

Scoping questions, in the order they pay off

Which other resources did this principal touch in the window? Which other principals did it create, assume, or modify — new users, new access keys, new roles, new trust policies? Does the source address or user agent appear anywhere else, in any account? Did anything persist — an EventBridge rule, a Lambda function, a launch template, an identity provider? And what data was read or copied, because that question drives the notification obligations rather than the technical response. Answer them in that order and the containment plan writes itself.

Correlating across services and accounts

No single service answers a scoping question on its own. Security Hub gives you every finding about the resource from every detector, deduplicated and in one schema, which is the fastest way to see that Inspector already reported an unpatched CVE on this instance. Detective gives you the relationships and the baselines over time. CloudTrail Lake or Athena gives you precise, citable queries. Amazon Security Lake gives you the normalised log data to run your own or third-party analytics across accounts. Use findings to orient, the graph to scope, and the logs to prove.

Key terms: Finding validation, Security incident (versus a finding), Amazon Detective, AWS CloudTrail

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.