Resource-level logging gaps
Lesson 2 of 3 in Troubleshooting Detection.
Here is the sentence this lesson exists to install: “no logs” is usually an IAM or resource-policy problem, not a logging problem.
AWS services almost never log on your behalf. They log as an identity you supply, into a destination that must independently agree to accept the write. So a per-service logging gap decomposes into the same three questions every time — is the feature switched on for this resource, which identity does the writing, and does the destination admit that identity — and the service-specific knowledge you need is simply where each service hides its answer to question two.
Lambda hides it in the function’s execution role. API Gateway hides it in an account-level setting you have to configure once per Region. CloudFront hides it in a bucket ACL requirement or a delivery-source configuration. Flow logs hide it in a delivery role whose errors appear only in the flow log’s own status field. Each is unremarkable on its own; together they are a large fraction of the troubleshooting questions on this exam.
| Service | Who writes the logs | Configured where | The gap looks like |
|---|---|---|---|
AWS Lambda | the function’s execution role | the role’s policy — | the function runs and returns correctly, and no log group or no stream ever appears |
Amazon API Gateway | an account-level IAM role trusted by |
| stage logging is switched on in the console and the log group stays empty |
Amazon CloudFront | the CloudFront log-delivery service | standard logging to S3, CloudWatch Logs or Firehose on the distribution; real-time logs to Kinesis with a sampling rate and field list | no log files appear, or only a sampled fraction of requests does |
VPC Flow Logs | a delivery role (to CloudWatch Logs) or the | the flow log resource itself — and the record format is fixed at creation | the flow log exists, no streams appear, and the error is only in the flow log’s status field |
Amazon Route 53 health checks | the health-check service; metrics publish to CloudWatch automatically | per health check — endpoint, calculated, or tracking a CloudWatch alarm | the check is “healthy” because nothing is watching the right thing, or nobody alarmed on it at all |
Amazon ECS / Amazon EKS | the task execution role (ECS) or the node/pod role (EKS) | the | containers run, stdout goes nowhere, and the cluster’s audit log type was never enabled |
Two chains, one symptom — Lambda and API Gateway
- Lambda function
Invoked, executes, returns a result. Logging failure changes none of this, which is why it goes unnoticed.
- Execution role
The identity Lambda uses to write logs. Needs logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents — AWSLambdaBasicExecutionRole is the managed shortcut. A permission boundary or SCP can remove these from a role that appears to have them.
- Log group /aws/lambda/<fn>
Created by Lambda on first write, with retention defaulting to never-expire. Pre-create it to control retention and encryption.
- API Gateway stage
Execution logging (debug the API) and access logging (who called it, in your chosen format) are two separate switches per stage.
- Account cloudWatchRoleArn
An IAM role trusted by apigateway.amazonaws.com with AmazonAPIGatewayPushToCloudWatchLogs, whose ARN is set on the API Gateway Account resource — once per Region. Missing this is the single most common API Gateway logging failure.
- Log group
Execution logs land in API-Gateway-Execution-Logs_
/ ; access logs go to the group you name. - Silence
Both chains fail the same way: the workload is healthy, nothing errors, and the evidence simply does not exist. This is why a Config rule and a zero-events alarm belong on every logging dependency.
API Gateway deserves its own paragraph because the exam asks about it in two different ways.
The account-level role. CloudWatch logging for REST APIs requires an IAM role whose trusted entity is apigateway.amazonaws.com, carrying the managed policy AmazonAPIGatewayPushToCloudWatchLogs, with its ARN set as cloudWatchRoleArn on the API Gateway Account resource — and you must set that property separately in every Region where you want logging. Because API Gateway calls AWS STS to assume the role, STS must be enabled in that Region too. Switching on stage logging without this is the classic “logging is enabled and the group is empty” scenario.
Execution versus access logging. They answer different questions and are enabled separately per stage. Execution logs are API Gateway debugging its own behaviour: which authorizer ran, how the request was mapped, what the integration returned, at an ERROR or INFO level you choose — and at INFO with full request and response data they can capture request bodies, which is a data-protection exposure worth deciding on deliberately rather than leaving on. Access logs are the who-called-my-API record, in a format you define with $context variables — requester identity, source IP, status, latency, authorizer result — and they are the ones a security team actually wants retained.
The pattern generalises. When a service offers two kinds of logging, one is almost always for debugging the service and one is for auditing the callers; the security answer is the second, and the first is where sensitive payloads leak into logs.
CloudFront: standard, real-time, and the thing people miss
Standard logging gives complete per-request records, delivered on a delay, to S3, CloudWatch Logs or Firehose. Real-time logs stream within seconds to Kinesis Data Streams with a configurable sampling rate and field list — so a question mentioning sampling, or seconds-level reaction at the edge, is pointing at real-time logs.
The thing people miss: origin logs are not a substitute. An ELB access log or an S3 server access log only sees requests CloudFront forwarded — cache hits and edge-blocked requests are invisible there. If the requirement is “all requests reaching our edge”, it has to be CloudFront’s own logging.
Route 53 health checks: what they prove and what they do not
A health check can watch an endpoint over HTTP, HTTPS or TCP from multiple AWS locations, be calculated from other health checks, or simply track a CloudWatch alarm — which means any metric you can alarm on becomes a health signal, and can drive DNS failover.
Two cautions the exam likes. A health check proves reachability, not correctness: an endpoint returning HTTP 200 with a broken dependency behind it is “healthy”. And the health-check endpoint is internet-reachable by design, so it must not expose version strings, internal hostnames or stack traces. Health-check status publishes CloudWatch metrics, so the actual detective control is an alarm on those metrics — a health check nobody alarms on is telemetry nobody reads.
ELB: access logs are off, and connection logs are separate
Load balancer access logs are disabled by default and enabled per load balancer, delivering to an S3 bucket whose policy must permit the delivery. Alongside the request fields they record the negotiated TLS protocol and cipher, which is the evidence behind weak-TLS findings. Application Load Balancers also offer connection logs as a separate feature covering TLS handshakes including those that failed — useful precisely because a failed handshake never becomes a request and therefore never appears in an access log.
Containers: the log driver and the role behind it
On ECS, container stdout reaches CloudWatch Logs through the awslogs log driver, and the write is performed using the task execution role — not the task role your application code uses. Mixing those two up produces a container that works perfectly and logs nothing. On EKS, application logs usually travel via Fluent Bit or the CloudWatch agent as a DaemonSet with an IAM role for service accounts, while control-plane log types (API server, audit, authenticator, controller manager, scheduler) are an explicit cluster setting — and the audit log type is the one a security team needs, enabled per cluster.
The general method, for a service this lesson did not name
You will meet services on the exam that are not in the table. Apply the same three questions: Is the feature on for this resource? (nearly everything is opt-in). Which identity performs the write? (a service principal, a delivery role, an execution role, a task execution role, an instance profile). Does the destination admit that identity? (bucket policy, logs resource policy, destination policy, KMS key policy). Then add the fourth question that catches the rest: is this Region the one I configured?
Key terms: Execution role, CloudWatch agent, VPC Flow Logs, Metric filter
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.