Catalogs and Lake Formation

Lesson 2 of 4 in Data Architecture for ML: Lakes, Warehouses, Catalogs.

A lake full of files is only data once something records what the files mean. The AWS Glue Data Catalog is that record — the metadata spine of the whole architecture: databases, tables, columns, formats, and partition lists describing objects in S3, compatible with the Hive metastore interface. Crawlers populate it by sampling the data, inferring schemas, and registering partitions as they appear. One catalog, every engine: Athena, Redshift Spectrum, EMR, Glue ETL — and the SageMaker Feature Store offline store registers its tables there too, which is why Athena can query your features at all.

Access is the catalog's other half. IAM and bucket policies stop at bucket and prefix granularity — useless when the requirement reads "the ML team may use the features but not the identity columns." AWS Lake Formation governs catalog resources centrally: grant at database, table, or column level, apply row-level security through data filters, and scale the whole scheme with LF-Tags — tag the PII columns once, grant "everything not tagged PII" everywhere. The engines that honor these grants are the usual suspects (Athena, Redshift Spectrum, EMR, Glue), so one permission model covers every road into the lake.

Key terms: Glue Data Catalog, AWS Lake Formation

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.