Lake, warehouse, or both

Lesson 1 of 4 in Data Architecture for ML: Lakes, Warehouses, Catalogs.

Domain 1 questions assume you know where data lives, so draw the two poles clearly. A data lake on S3 stores anything — Parquet, CSV, images, audio, raw logs — with the structure applied when you read (schema-on-read). Every engine in this domain speaks to it: Glue, Athena, EMR, SageMaker, even Redshift. It is ML's natural home for two blunt reasons: training jobs consume files from S3, and multimodal data fits nowhere else. A data warehouse — Amazon Redshift — is the opposite bargain: structure enforced on write (schema-on-write), columnar MPP storage, and in exchange, fast complex SQL for BI dashboards and analysts.

Lake, warehouse, and the pattern between them
S3 data lakeRedshift warehouseLakehouse pattern

Schema

On read — store first, interpret later

On write — modeled up front

Raw stays on-read; curated marts on-write

Data shapes

Anything, including images, audio, documents

Structured tables

Both, each where it belongs

Primary consumers

ML training, Spark, ad-hoc Athena SQL

BI dashboards, analysts at SQL speed

Each tool at its own layer

ML fit

Training channels read S3 natively

Redshift ML trains from SQL; otherwise unload to S3 for SageMaker

Train from the lake; serve hot aggregates from the warehouse

The "both" options are where exam points hide. Amazon Athena runs serverless SQL directly on S3 — no loading, no infrastructure, pay per query. Redshift Spectrum lets a Redshift cluster query S3 external tables (registered in the Glue Data Catalog) alongside its local tables — the warehouse reaches into the lake. The lakehouse pattern combines them deliberately: raw and feature data stay in S3 where training reads them; curated, hot aggregates load into Redshift for the dashboard crowd.

Lake, warehouse, or both?

Interactive decision tree — outcomes:

  • S3 data lake

    Store first, interpret on read: any format — Parquet, images, audio, logs — and SageMaker training channels consume it natively. The lowest-cost home for training data at scale.

  • S3 lake + Athena

    Keep the lake as-is and point Athena at it: serverless SQL, no loading, no infrastructure, pay per query — the "SQL on S3 without moving it" answer when no warehouse exists.

  • Amazon Redshift

    Schema-on-write, columnar MPP: model the tables up front and buy fast, complex SQL for dashboards and analysts in exchange.

  • Redshift + Spectrum

    External tables registered in the Glue Data Catalog join local Redshift tables in one query — the warehouse reaches into the lake, and nothing is loaded or duplicated.

  • The lakehouse pattern

    Deliberately both: raw and feature data stay in S3 where training jobs read them; curated aggregates load into Redshift for dashboard SQL. Each consumer gets the store built for its workload.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.