Querying at scale

Which query engine?

Interactive decision tree — outcomes:

• CloudWatch Logs Insights

  Query multiple log groups at once, auto-discovered fields on JSON events, stats … by aggregation, parse for unstructured lines. Charged per gigabyte scanned, so set the time range first. The fastest path from “something just happened” to an answer.

• A metric filter and an alarm, not a query

  If the same query is going to be run forever, it is not a query — it is a detective control. Convert it into a metric filter plus an alarm, or a subscription filter into something that reacts. Repeatedly re-running an interactive query is a cost pattern, not a monitoring design.

• You are querying the wrong copy

  If the question regularly spans months, CloudWatch Logs is the expensive place to hold the answer. Subscribe the log group to Firehose, land it in S3 as Parquet, and query with Athena — keeping CloudWatch retention short for triage. This is the architecture fix an exam scenario is usually hinting at when it mentions both cost and long history.

• Amazon Athena

  Serverless SQL over S3, charged per terabyte scanned. Partition by date, account and Region; store as Parquet; use partition projection so there is no partition-management job. The default engine for CloudTrail, flow logs and Security Lake history.

• Amazon OpenSearch Service

  Indexed, sub-second, full-text, with dashboards and its own alerting. The cost is that indexed storage is the expensive tier and the index must be fed and sized — so use it for a deliberate hot window, not for everything you own.

• Security Lake first, then Athena

  The problem is not the engine, it is the schema. Normalize to OCSF through Security Lake and the cross-source query becomes a single SQL statement instead of five parsers. Athena is then the query subscriber.

• Amazon Managed Grafana

  A managed workspace pointed at several data sources — CloudWatch, OpenSearch, Athena, Prometheus — so one dashboard holds alarm state, a log panel and a SQL result side by side. Visualization and correlation, with IAM Identity Center sign-in. It stores nothing itself.

Logs Insights

Find the top source addresses producing AccessDenied in the last hour, across several log groups:

fields @timestamp, userIdentity.arn, sourceIPAddress, eventName
| filter errorCode = "AccessDenied"
| stats count(*) as denials by sourceIPAddress, userIdentity.arn
| sort denials desc
| limit 20

Fields in JSON events are discovered automatically, so userIdentity.arn works with no schema definition. parse handles unstructured lines when there is no JSON to discover. The time range is the cost control — you are charged for bytes scanned, and the selector above the query is what decides how many.

Athena

The same question over a quarter of CloudTrail history in S3:

SELECT sourceipaddress, useridentity.arn, count(*) AS denials
FROM   cloudtrail_logs
WHERE  errorcode = 'AccessDenied'
  AND  account = '111122223333'
  AND  region  = 'eu-west-1'
  AND  date BETWEEN '2026/07/01' AND '2026/09/30'
GROUP  BY 1, 2
ORDER  BY denials DESC
LIMIT  20;

The three predicates on account, region and date are not filters for readability — they are partition pruning, and they are the difference between scanning gigabytes and scanning the whole archive. Selecting named columns rather than * is the second lever, and only pays off on columnar Parquet.

OpenSearch

Where analysts need to explore rather than aggregate — “show me everything mentioning this hostname, then pivot to what else that user did” — an index earns its cost:

{ "query": { "bool": {
    "must":   [ { "match": { "message": "AccessDenied" } } ],
    "filter": [ { "range": { "@timestamp": { "gte": "now-24h" } } } ] } },
  "aggs": { "by_ip": { "terms": { "field": "sourceIPAddress", "size": 20 } } } }

Fed by a subscription filter or Firehose. Full-text search and interactive pivoting are what you are paying the index for; if every question you ask is a date-ranged aggregate, Athena does it cheaper.

Managed Grafana

For the standing view rather than the one-off question, a Grafana workspace holds several data sources on one dashboard:

• a CloudWatch panel for alarm state and the metric-filter counters
• an OpenSearch panel for the last 24 hours of matching log lines
• an Athena panel for the 90-day trend of the same signal
• annotations marking deploys and incidents across all three

The value is the single pane during an incident, not the query language. Access is through IAM Identity Center or SAML with Grafana roles, and a service role lets the workspace reach data sources in other accounts. Grafana stores no log data of its own — every panel is a live query against something else, so every panel has a cost.