CloudWatch Logs architecture
Lesson 2 of 4 in The Logging Architecture.
AWS services publish to CloudWatch Logs on their own. Your instances do not. Nothing inside an EC2 instance, a container host or an on-premises server reaches CloudWatch Logs until the CloudWatch agent is there to send it — and that single sentence accounts for a large share of the “missing logs” scenarios in this exam.
Four conditions must all hold, and each one is a scenario in its own right:
- Installed. Baked into the AMI, installed by a Systems Manager document, or deployed as an EKS add-on.
- Running. A crashed or stopped agent is the quietest failure in the stack: the instance is healthy, the application is serving traffic, and the log group simply stops gaining events.
- Correctly configured. The agent configuration file names the log file paths, the destination log group and log stream, and the Region. A path typo, an application that rotates to a name the glob does not match, or a log group name that differs by one character all produce exactly the same symptom — no data, no error.
- Permitted. The instance profile must allow writing to CloudWatch Logs;
CloudWatchAgentServerPolicyis the managed shortcut. An instance with no role, or a role whose permissions were tightened by a boundary, cannot deliver.
The operational answer to all four is to stop treating the configuration as a file on a host. Keep it in SSM Parameter Store, apply it with a State Manager association, and let every new instance inherit it on a schedule rather than waiting for someone to remember.
One log group, two exits
- Instance / container
Application and OS log files on disk. Invisible to AWS until an agent reads them.
- CloudWatch agent
Reads the configured paths, publishes to a named log group and stream, and also sends memory, disk and process metrics the hypervisor cannot see.
- Log group
The retention boundary (1 day to 10 years, or never expire), the KMS encryption boundary, and the unit metric and subscription filters attach to. Streams within it are per source.
- Metric filter
A pattern that increments a CloudWatch metric on each match — e.g. { $.errorCode = "*UnauthorizedOperation" }. Counts; does not notify.
- CloudWatch alarm
Watches the filter metric and changes state. THIS is what notifies, via SNS or an EventBridge rule. A filter with no alarm is a metric nobody reads.
- Subscription filter
A near-real-time feed of matching events out of the log group. Forwards rather than counts.
- Firehose → S3
The cheap archive-and-query path: buffer, optionally transform with Lambda, convert to Parquet, land in the log archive for Athena.
- Lambda
Transform, enrich, normalize to OCSF, or forward to a third-party SIEM endpoint.
- OpenSearch Service
The interactive-search path: indexed, searchable, dashboards and alerting over a recent hot window.
Two properties of the left-hand exit are reliably tested. A metric filter applies only to events ingested after it is created — filters are never retroactive, so creating one today tells you nothing about last week. And the filter itself never notifies: it publishes a metric, and the alarm on that metric is what pages someone. “We created a metric filter for root sign-ins but nobody was alerted” is a missing-alarm problem.
The classic detective controls are all built this way: alarm on root account usage, on IAM policy changes, on console sign-in failures, on security group changes, on CloudTrail configuration changes, on KMS key deletion scheduling. Patterns can be JSON selectors for structured events ({ $.eventName = "StopLogging" }) or plain term matches for unstructured text.
Retention is set per log group, from one day to ten years or never-expire, and the default on a hand-created group is never-expire — which is how accounts accumulate years of debug logs at full price. Retention is also a security decision in both directions: too short and the evidence is gone before an investigation starts (a breach discovered 120 days later against a 30-day retention is an unanswerable question), too long and you are holding data you promised to delete. Encrypt the group with a customer managed KMS key where the logs carry sensitive content, and note that the key policy must allow the logs service principal for the Region.
| Destination | Shape | Reach for it when… | Watch out for |
|---|---|---|---|
Amazon Data Firehose | buffered delivery to S3, OpenSearch, or an HTTP endpoint | you want durable, cheap archive in the log bucket — optionally Parquet for Athena | buffering adds latency by design; not the path for a minute-level alert |
AWS Lambda | your code, invoked with batches of events | you need to transform, enrich, deduplicate, or normalize to OCSF before storage | you now own retries, concurrency limits and a dead-letter path |
OpenSearch Service | indexed documents, searchable and dashboarded | analysts need interactive full-text search over a recent window | indexed storage is the expensive tier — size the hot window deliberately |
Kinesis Data Streams | an ordered, replayable stream with multiple consumers | several independent consumers need the same feed, or you need replay | shard capacity is yours to manage; throttling shows up as dropped delivery |
Cross-account destination | a destination resource in the logging account wrapping one of the above | every workload account must ship logs to one central logging account | needs a destination policy in the receiver and the right permissions in each sender |
That last row is the design the exam guide names outright: a dedicated CloudWatch logging account. The shape is the same as the CloudTrail one and for the same reason — evidence should not live where the incident happens. Each workload account’s log groups carry subscription filters pointed at a destination in the logging account; the destination has a resource policy naming the permitted sender accounts or the organization, and it fronts a Kinesis stream or Firehose that lands everything centrally.
CloudWatch also supports cross-account observability, where monitoring accounts are linked to source accounts and can read their log groups, metrics and traces directly in the console without copying anything. The two approaches answer different questions, and exam wording distinguishes them: cross-account observability is for looking at logs where they are; subscription filters to a central destination are for collecting logs you intend to retain, normalize and query as one corpus. A SOC usually wants both.
One honest limit worth knowing: a log group supports only a small number of subscription filters, so “subscribe the group to four different tools” hits a ceiling. The scalable pattern is one subscription into a stream or Firehose, and fan-out downstream.
Key terms: CloudWatch agent, Metric filter, Subscription filter, CloudWatch Logs data protection policy
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.