Why the logs are missing

The missing-logs checklist, in order

Interactive decision tree — outcomes:

• It was never on

  By far the most common answer, and the one with no remedy for the past. Data events are off by default on every trail. Flow logs, Resolver query logs, ELB access logs, CloudFront logging, API Gateway stage logging and S3 server access logging are all opt-in, per resource. Enable it, then enforce it with a conformance pack so the next resource is not a repeat.

• The writer cannot write

  A flow log needs a delivery role that CloudWatch Logs trusts. A Lambda function needs logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents in its execution role. An instance needs log-write permissions in its instance profile. A permission boundary or SCP can also be removing an action the role appears to have.

• The destination refuses the delivery

  The bucket policy must name the service principal (cloudtrail.amazonaws.com, delivery.logs.amazonaws.com) with the right actions and prefixes, and usually a source-ARN or source-account condition. A cross-account CloudWatch Logs destination needs a destination policy naming the permitted sender accounts or the organization. Resource policies are the half of cross-account access people check second.

• The producer cannot encrypt

  If the destination is encrypted with a customer managed key, the key policy must allow the producing service — kms:GenerateDataKey* for CloudTrail or the logs service principal for an encrypted log group, scoped with a source condition. Get this wrong and delivery stops with no error visible on the trail or log group.

• The reader cannot decrypt

  The objects are there. Reading an SSE-KMS object needs s3:GetObject and kms:Decrypt, and the key policy is the authority on the second. The failure surfaces as an S3 AccessDenied, which is why this one costs an hour in the bucket policy before anyone opens the key.

• Four conditions, one symptom

  The CloudWatch agent must be installed, running, configured with the correct paths, log group, stream and Region, and permitted by the instance profile. A stream that silently stops on a healthy instance is the signature. Fix it once with a State Manager association and the configuration stops being a per-host artifact.

• You are looking in the wrong Region

  A single-Region trail records nothing elsewhere. The API Gateway CloudWatch role is an account setting per Region. Log groups, flow logs and destinations are all regional. Global-service events need a multi-Region trail. The cheapest check on this list, and the one skipped most often.

• Something upstream is dropping records

  A CloudTrail advanced event selector scoped to one bucket records nothing about another. A subscription filter pattern forwards only matches. CloudFront real-time logs ship a sampling percentage. A WAF logging configuration can redact fields. Partial data is a filter problem, not a delivery problem — and the shape of what did arrive tells you which filter.

• They arrived, and you deleted them

  A log group’s retention, an S3 lifecycle expiration, or a lifecycle rule expiring non-current versions. This is a retention-design failure rather than a logging failure — and the reason an incident discovered 120 days later against 30-day retention is an unanswerable question. Re-read the archive design in the logging architecture module.

• Then the event never happened in that log’s field of view

  The honest last answer. Flow logs do not record traffic to the Amazon DNS resolver. CloudTrail does not record a packet. WAF logs do not record requests that never reached the distribution. Before concluding that logging is broken, confirm the log you are reading is capable of containing the event — that is a coverage question, and the fix is a different log source.

Symptom: a Lambda function produces no logs at all

Cause: the function’s execution role lacks logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents — or the log group exists with a resource policy or encryption key that denies the role. Lambda does not log on your behalf; it logs as your function’s role.

Fix: attach AWSLambdaBasicExecutionRole or the equivalent inline statements, and check for a permission boundary or SCP stripping the action. Pre-create the log group with the retention you want, because the one Lambda creates defaults to never-expire.

Symptom: API Gateway stage logging is enabled but the log group stays empty

Cause: the account-level CloudWatch role is not set. API Gateway needs an IAM role trusted by apigateway.amazonaws.com — the managed policy AmazonAPIGatewayPushToCloudWatchLogs covers it — and that role’s ARN must be set as cloudWatchRoleArn on the API Gateway Account resource, separately in each Region.

Fix: create the role, set cloudWatchRoleArn for that Region, and only then expect stage logging to produce anything. Note the two log types are separate switches: execution logging for debugging the API’s own behaviour, access logging for who called it, with a format you define.

Symptom: flow logs were created but the log group has no streams

Cause: usually the delivery role. A flow log publishing to CloudWatch Logs needs an IAM role that the flow-logs service can assume with permission to create streams and put events; publishing to S3 instead needs the bucket policy to admit delivery.logs.amazonaws.com.

Fix: check the flow log’s own status field — it reports delivery errors there rather than anywhere obvious — then the role trust policy, then the destination policy. And remember there is simply no data until there is traffic on the monitored interface.

Symptom: GuardDuty findings stopped appearing for some accounts

Cause: not a logging problem. Either those accounts were created after the organization rollout and auto-enable for new accounts was never switched on, or a suppression rule is archiving the findings, or a trusted IP list is preventing them being raised, or a protection plan covering that resource type is off in those accounts.

Fix: check the delegated administrator’s account list and auto-enable setting first — it explains the “exactly the newest team” pattern that the others do not.

Symptom: the alarm never fired even though the log line is clearly there

Cause: one of three. The metric filter was created after the event — filters are never retroactive. The filter publishes a metric but no alarm was built on it. Or the alarm treats missing data as notBreaching, so a metric that only exists when something bad happens never leaves INSUFFICIENT_DATA.

Fix: build the alarm, set the missing-data behaviour deliberately, and test it by generating the condition on purpose. An untested detective control is a hypothesis.

Symptom: CloudTrail shows the API call but not what was in it

Cause: working as designed. CloudTrail records request parameters, but sensitive values are omitted and large request bodies are not stored; some fields are redacted by the service. CloudTrail is an audit record of calls, not a proxy capture.

Fix: nothing in CloudTrail. If you need request content, that is an application-level log, an access log, or WAF logging — and each has its own data-protection consequences, since request content is exactly where secrets live.