The logs CloudTrail does not have
Lesson 2 of 4 in Audit Logs and Privacy.
CloudTrail records AWS API calls. A great deal of what happens in a data platform is not an AWS API call, and for that there is a second family of logs — service-native, each opt-in in its own way, each answering a question CloudTrail structurally cannot.
Amazon Redshift. A SELECT inside the warehouse is SQL, not an API call, so CloudTrail never sees it. Database audit logging fills the gap with three logs, and the enabling sequence is the examinable part. The connection log records authentication attempts, connections and disconnections with client address and authentication method; the user log records changes to database user definitions; the user activity log records every query before it runs. All three are switched on together — and the user activity log additionally requires the database parameter enable_user_activity_logging to be true, which is false by default. Enable the feature without the parameter and you get connections and user changes and not one query, which is the single most common half-configured audit on AWS. Parameter-group changes take effect on reboot. Provisioned clusters can export to Amazon S3 or CloudWatch Logs; Redshift Serverless exports to CloudWatch only.
Separately and always on, Redshift logs to its own system tables and views — STL_QUERY, SYS_QUERY_HISTORY, STL_CONNECTION_LOG. No configuration, immediately queryable with SQL, and retained for only a few days. So system views answer today's question and audit logging answers the auditor's; neither replaces the other.
Amazon Athena. CloudTrail records that StartQueryExecution happened and by whom. The statement text and the bytes scanned live in query history, retrievable with ListQueryExecutions and GetQueryExecution. Retention is limited, so a requirement to keep query history for an audit period means exporting it. The division of labour on a governed lake is worth holding as a sentence: Athena history says what was asked, GetDataAccess says what was authorised, S3 data events say what bytes moved.
Amazon S3 server access logs. Server access logging records every request to a bucket — requester, key, operation, bytes sent, HTTP status, turnaround time. It looks like a cheaper substitute for data events and is not one. AWS is unusually blunt about why: delivery is best effort, records arrive within a few hours, "the completeness and timeliness of server logging is not guaranteed", a record "might not be delivered at all", and you may even see a duplicate. So it is genuinely good for access-pattern analysis and cost attribution, and unusable where an auditor needs a count they can rely on — data events are near-real-time, delivered to a trail, and can be integrity-validated, which server logs cannot.
The comparison is not one-directional, though, and two differences run the other way. Server access logs record authentication failures, which CloudTrail data events do not, and they record lifecycle transitions, expirations and restores — which is how you demonstrate that a retention rule actually ran. CloudTrail, in exchange, can be scoped to a prefix and carries full identity context. The design that follows is to use both: data events for identity and evidence, server logs for HTTP-level detail, failed requests and lifecycle activity.
AWS Glue writes job logs to CloudWatch Logs, with continuous logging streaming driver and executor output during the run rather than only at the end — the difference between debugging a two-hour job live and waiting for it to fail. Job run history, including arguments and status, is retrievable from the Glue API. Amazon EMR writes step and application logs to Amazon S3 and optionally to CloudWatch, and a cluster's logs survive its termination only if the log URI was set at launch, which is a decision you cannot revisit. Lake Formation surfaces recent GetDataAccess activity on its own console dashboard, which is the fastest way to answer "who has been querying this table lately" without writing a query.
| Log | On by default? | Answers | Cannot answer |
|---|---|---|---|
Yes | Who changed a schema, a grant, a bucket policy, a cluster setting — and who submitted a query | Whether any data was read, or what a query asked for | |
No — and charged per event | Which objects or items were read and written, near real time, integrity-validated | On a governed lake, which person — the principal is the registration role unless source identity is on | |
No — storage cost only | HTTP-level detail, authentication failures, and lifecycle transitions and expirations — none of which CloudTrail data events carry | Nothing with guaranteed completeness — delivery is best effort, within a few hours, and a record may be missing or duplicated | |
No — and the query log needs a parameter change plus a reboot | Connections and authentication; user definition changes; every query, with | Nothing about data read through Redshift Spectrum from the lake — that is | |
Yes — and not configurable | What ran recently, immediately, with a | Anything older than a few days — retention is a short rolling window | |
Yes, for a limited window | The statement text, the workgroup, the output location, the bytes scanned | A retained record — it must be exported if the audit period is months | |
Yes — it is a management event | Which principal was authorised for which table, through which engine | Which rows or objects were actually returned |
Key terms: Redshift database audit logging, Redshift STL and SYS system views, Athena query history, S3 server access logging, CloudWatch Logs Insights
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.