OpenSearch for search and logs
Lesson 4 of 5 in Analyzing Data.
OpenSearch Service earns its place in a data platform for one capability the rest of this module does not have: an index. Athena and Redshift Spectrum scan; OpenSearch looks a term up. That difference is worth paying for when the question is exploratory — "show me everything mentioning this order id, now pivot to what else that account did" — and worth nothing when the question is a date-ranged aggregate, which Athena answers more cheaply.
So the decision is not "search or SQL", it is how much of your history deserves to be indexed. Indexing costs storage and compute continuously, for every document, whether anybody searches it or not. Scanning costs nothing until somebody asks.
Which makes the real design question a tiering question, and OpenSearch has a three-tier answer:
- Hot — attached storage on data nodes, the only tier you can write to, lowest-latency search.
- UltraWarm — read-only, backed by S3 with aggressive local caching, scaling to petabytes at a fraction of hot cost, still interactively searchable.
- Cold — detached from compute entirely, unlimited archival, and not searchable until migrated back to UltraWarm.
And the honest fourth option, which a scenario will sometimes be steering you toward: none of the above. If a year of logs is only ever read by a quarterly compliance query, S3 plus Athena is cheaper than any OpenSearch tier, and the only thing you give up is interactive search you were not using.
| Need | Right tool | Why the other one loses |
|---|---|---|
Free-text search and interactive pivoting over the last two weeks | OpenSearch, hot plus UltraWarm | Athena would scan the whole window for every exploratory question, and exploration means many questions |
A daily aggregate over three years of history | Athena over partitioned Parquet | indexing three years costs continuously to answer one scheduled query a day |
Sub-second alerting on a pattern in arriving records | OpenSearch alerting, or a CloudWatch metric filter | a scan-based engine cannot meet a sub-second target, and re-running a query forever is a monitoring design smell |
Seven-year retention for an occasional audit request | S3 with lifecycle rules, queried by Athena | even cold storage pays for a domain; an S3 prefix costs storage alone and needs no migration to be read |
Key terms: Index State Management, UltraWarm, Amazon OpenSearch Service, Amazon Athena
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.