Centralising logs for an audit

Lesson 3 of 4 in Audit Logs and Privacy.

Logs scattered across accounts, Regions and services are evidence nobody can use. An audit-ready design has four properties, and each one answers a specific objection an auditor will raise.

Complete. The coverage question is "how do you know nothing is missing", and the answer is architectural rather than procedural. An organization trail is created once in the management or a delegated administrator account and applies to every member account — including accounts added afterwards, which is the property a per-account trail can never have, because a new account arrives with no trail until someone remembers. Make it multi-Region at the same time, since a single-Region trail misses activity in every Region nobody was watching. Pair it with an AWS Config rule reporting buckets without logging and clusters without audit logging, so coverage is a report rather than a belief.

Tamper-evident. Log file validation makes CloudTrail write a signed digest file per hour, hash-chained to the previous one, so a deleted or modified log file is detectable and provable. The destination belongs in a dedicated log archive account that no production principal can write to or delete from, with Object Lock in compliance mode on the archive so that even an administrator cannot shorten history, and a service control policy denying cloudtrail:StopLogging and cloudtrail:DeleteTrail organization-wide. The threat model is not a disk failure; it is a privileged person removing the record of what they did.

Queryable. Raw JSON in Amazon S3 is storage, not analysis. Three routes, and the choice is about retention and effort. Athena over the trail bucket is the cheapest: define an external table and use partition projection so the account, Region and date partitions are computed from the key layout rather than crawled, which matters because a trail bucket accumulates partitions faster than any crawler enjoys. CloudTrail Lake is managed: an event data store with SQL queries, configurable retention up to ten years, and no table to maintain. Security Lake normalises logs from many sources into the OCSF schema in Parquet and is the right answer when the requirement is to correlate CloudTrail with VPC flow logs, Route 53 logs and third-party findings in one query.

Retained for exactly as long as required. CloudWatch Logs groups default to never expiring, which is a cost problem and a privacy problem at once. Set a retention period per group deliberately. On the archive side, S3 Lifecycle moves old logs to colder classes and expires them on a schedule — and remember the precedence: a lifecycle expiration cannot delete an object version still under an Object Lock retention period, which is exactly why the two combine safely. Object Lock holds the record for the required term and the lifecycle rule expires it shortly after, so the obligation ends on schedule rather than becoming a permanent archive.

The wider architecture — detection pipelines, alerting, log immutability design and the multi-account layout — is taught on the security path at logging architecture and retention, immutability, and protecting the evidence.

From a scattered estate to evidence an auditor accepts

  1. Start from the question, not the log

    Write down the three or four questions you will actually be asked — "who read this object", "who changed this grant", "who ran this query", "when was this record deleted" — and work backwards. Collecting everything and hoping is how organisations pay for petabytes that answer none of them.

  2. One organization trail, multi-Region

    Created once, covers every member account including future ones, and member accounts cannot stop or delete it. This is the coverage guarantee; nothing procedural substitutes for it. Management events arrive by default — data events still have to be chosen, and on an organization trail that choice applies organization-wide.

  3. Decide data events deliberately, per bucket

    The expensive decision, and the one that cannot be made retrospectively. Use advanced event selectors to scope by resource ARN prefix so the sensitive prefixes get full read/write logging while a bulk staging prefix does not. AWS’s own guidance: for auditing, log all data events on the resources that matter.

  4. Add the service-native logs

    Redshift audit logging with enable_user_activity_logging and a reboot; Athena query history exported if the audit period exceeds its retention; Glue job logs with continuous logging; EMR log URI set at launch, because it cannot be added later. These are the questions CloudTrail structurally cannot answer.

  5. Land it all in a log archive account

    A separate account with no production workloads, a tiny set of principals, and no human delete permission. Production can write and cannot remove. Add Object Lock in compliance mode on the archive and an SCP denying cloudtrail:StopLogging and DeleteTrail across the organization.

  6. Turn on log file validation

    Signed hourly digests, hash-chained, so a removed or altered file is provably detectable. Cheap, usually forgotten, and the difference between "we have logs" and "we can show these logs are the ones we wrote".

  7. Make it queryable before you need to query it

    Athena with partition projection for the cheapest path over the trail bucket; CloudTrail Lake when you want managed retention up to ten years and no table to maintain; Security Lake when several log sources must be correlated in one OCSF schema. Write and save the queries for your four questions now, while nothing is on fire.

  8. Set retention on both ends, and test the restore

    CloudWatch Logs groups never expire by default — set it. On the archive, pair Object Lock with a lifecycle expiration so the record ends on schedule instead of forever. Then run one of your saved queries against six-month-old data: a log you cannot query is not evidence, and finding that out during an audit is the outcome this whole design exists to avoid.

Key terms: Log archive account, Log file integrity validation, AWS CloudTrail Lake, Amazon Security Lake, S3 Object Lock, Evidence (Audit Manager)

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.