Normalizing and correlating
Lesson 3 of 3 in Log Analysis at Scale.
Every real investigation ends up asking for the same artifact: one timeline. At 02:14 a role was assumed from an unfamiliar address. At 02:16 an instance profile’s credentials were used to list buckets. At 02:19 an outbound flow of 2 GB began. At 02:31 GuardDuty raised a finding. Four facts, four different log sources, one story — and the story only exists if something stitched them together.
Stitching needs three things, and each one is a design decision rather than a tool:
A common schema. The fields have to mean the same thing across sources, or every join is a guess. This is what OCSF buys you, and it is why normalization comes before correlation rather than after it.
A common clock. Timestamps must be in one timezone — UTC, always — at a consistent precision, and you must know each source’s delivery lag as distinct from its event time. A flow log aggregates over a one- or ten-minute interval, so its timestamps are a window, not an instant; a CloudTrail event has a precise event time but may arrive minutes later. Ordering by arrival rather than by event time produces a confident, wrong timeline.
Correlation keys. Something must be shared between records: an IP address, an instance id, a role session name, a request id, an account id. Choosing those keys up front is what makes the join possible — and it is the reason the pkt-srcaddr field and a sensible role-session-naming convention are security decisions, not operational trivia.
The transformer does five jobs, and naming them separately is what keeps a pipeline maintainable: parse the source format, normalize field names and types to one schema, enrich with context the log lacks, de-duplicate against a stable key, and route — not everything belongs in the same place. High-value, low-volume findings can go to a hot index and the archive; verbose debug records can go to the archive only.
Two things about third-party integration are worth stating because the exam guide names it explicitly. First, the integration surface is usually one of three shapes: a Security Lake data-access subscriber (notified of new objects, reads the Parquet), a Firehose HTTP endpoint delivering to the vendor’s ingest URL, or Security Hub receiving the vendor’s findings so they join your single list. Second, the schema is the contract — a vendor that reads OCSF needs no bespoke parser from you, and that is the commercial argument for normalizing before you integrate rather than after.
And know when not to build this. If the requirement is “correlate AWS sources into one queryable store”, Security Lake already is this pipeline, with the ETL, the schema and the subscriber model included. Hand-rolling Firehose and Lambda is the right answer when you need a transform Security Lake does not perform — a proprietary on-premises format, an enrichment that must happen at write time, a routing rule specific to your organization. “Build the pipeline” and “use the managed lake” are both correct answers to different questions, and the distinguishing phrase is usually whether non-AWS sources in an unsupported format are involved.
Clock skew and the two different times
Every record has at least two timestamps: when the event happened and when it arrived. Order by arrival and your timeline lies — a CloudTrail event delivered five minutes late will appear after the flow log it actually preceded. Normalize everything to UTC at a consistent precision, keep both fields, order by event time, and record each source’s typical delivery lag so an analyst knows how long to wait before concluding that something is absent. Flow log timestamps are a one- or ten-minute aggregation window, not an instant, and treating a window as a point is a classic false conclusion.
Identity resolution: the same actor under four names
CloudTrail identifies a principal by ARN and sometimes by access key id. A flow log identifies a workload by interface id. A GuardDuty finding names an instance id. An application log has a username your IdP issued. Joining them requires a resolution step — interface to instance to instance profile to role ARN — and the practical fix is to enrich at write time with the identifiers you will want to join on, plus a disciplined RoleSessionName convention so an assumed-role session traces back to a human or a workload rather than to a shrug.
Duplicates, and why they are the default
A trail delivering to S3 and CloudWatch Logs produces two copies. Streaming delivery is at-least-once, so a retried batch produces more. A subscription filter and a Firehose on the same log group produce two paths to the same store. De-duplicate on a stable key — source, event id and event time — rather than hashing the whole record, which changes whenever you add an enrichment field. Duplicates do not merely inflate storage: they double counts in exactly the aggregate queries an investigator is relying on.
Cardinality and cost, the quiet pipeline killers
Enrichment that adds a high-cardinality field to every record — a request id as a partition key, say — multiplies small files and wrecks query planning. Keep partitions coarse (account, Region, date) and put high-cardinality values in columns, not in the prefix. Likewise, enriching every verbose debug record with a lookup is a Lambda bill with no investigative value; enrich the sources an investigation actually joins on, and let the rest land raw.
Key terms: Open Cybersecurity Schema Framework (OCSF), Amazon Security Lake, Amazon Data Firehose, Amazon Managed Grafana
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.