The analytics yard: Glue, Athena, EMR, Redshift
Lesson 2 of 4 in The Supporting Cast: Data, Compute, and Pipes.
Between "data exists" and "data is trainable" sits the analytics yard. The busiest corner is the Glue family: Glue ETL runs serverless Spark transform jobs; crawlers scan S3 and populate the Glue Data Catalog — the shared metastore that Athena, EMR, Redshift Spectrum, and Lake Formation all read; DataBrew does visual, no-code preparation; and Glue Data Quality evaluates rules that can gate a pipeline. The rest of the yard divides by verb: query, process, warehouse, stream, search, govern, visualize.
| Service | What it is | The scenario says… |
|---|---|---|
AWS Glue | Serverless ETL (Spark) + crawlers + the Data Catalog | "catalog and crawl data in S3", "serverless transform before training" |
Amazon Athena | Serverless SQL directly over S3 | "ad-hoc SQL on the lake", "analysts explore training data without a cluster" |
Amazon EMR | Managed big-data clusters (Spark and the Hadoop ecosystem) | "existing Spark jobs", "petabyte-scale processing with cluster-level control" |
Amazon Redshift | The data warehouse (provisioned or serverless) | "BI-grade SQL at scale", "warehouse joins feeding feature queries" |
Kinesis Data Streams | Real-time event ingestion with replayable retention | "thousands of events per second", "multiple consumers re-read the stream" |
Amazon Data Firehose | No-code delivery of streams into S3, Redshift, OpenSearch… | "easiest way to land streaming data in S3", "near-real-time, no consumer code" |
Managed Service for Apache Flink | Stateful stream processing (the service formerly named Kinesis Data Analytics) | "join two streams", "exactly-once", "windowed aggregation in flight" |
OpenSearch Service | Search and analytics engine, including k-NN vector search | "full-text search over logs/documents", "vector store for RAG" |
Lake Formation | Permissions layer for the lake: table- and column-level grants on the Data Catalog | "analysts may query the lake but not see PII columns" |
Amazon Quick | The BI suite that succeeded QuickSight — dashboards and visual analysis | "business dashboards over Athena/Redshift" (older material may still say QuickSight) |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.