Lake Formation and DataZone
Lesson 4 of 4 in Cataloging and Discovery.
There are two ways to decide who may read a table in the Data Catalog, and almost every confusing permission incident in a lake comes from a table being governed by one of them while somebody believes it is governed by the other.
The default model is IAM plus storage permissions, evaluated separately. A principal needs catalog permissions to resolve the table and s3:GetObject on its location to read the bytes. The query engine reads S3 with the caller's own credentials. This works, and it has a shape you cannot escape: S3 permissions are expressed over buckets and prefixes, so the finest control you have is "this prefix". There is no way to say these four columns or only the rows where region is EU, because S3 has no idea what a column is.
AWS Lake Formation moves the decision up to the table. You register the S3 location with Lake Formation, grant SELECT to a principal on a table — optionally restricted to named columns, to rows matching a predicate, or to both — and then take the principal's direct S3 permission away. At query time Lake Formation checks the grant and hands the engine temporary credentials scoped to exactly the permitted data, plus any filters for the engine to apply. That is credential vending, and it is what makes column-level and row-level control real rather than aspirational.
The important corollary: a directly attached s3:GetObject on the same prefix bypasses the whole model. It is not a convenience, it is a hole, and finding one is a normal outcome of auditing a lake that adopted Lake Formation without finishing the job. This lesson covers the permission model as it applies to the catalog; the identities, roles and federation on the other side of it are covered in authentication and authorization.
| State | Who authorises the data read | Finest grain | Recognise it by |
|---|---|---|---|
Glue resource (not registered) | the caller’s own IAM and bucket policies | the S3 prefix — no column or row control is expressible |
|
Lake Formation resource | Lake Formation, which vends temporary scoped credentials to the engine | the cell — named columns, a row predicate, or both together | the location is registered and |
Hybrid resource | Lake Formation for opted-in principals, IAM for everyone else | the cell for opted-in principals, the prefix for the rest | registered in hybrid mode; the flag is true only for principals who opted in |
Three capabilities sit on top of the permission model, and each answers a question about scale rather than about mechanism.
Data filters are named rules that narrow a grant. Column-level filtering includes or excludes named columns; row-level filtering carries a predicate such as region = 'EU'; together they give cell-level access. The engine applies them on Lake Formation's instruction, so the analyst runs an ordinary SELECT * and receives only permitted cells. The usual exam shape is one table that several teams must read with different visibility, where the wrong answers are "make a filtered copy per team" and "make a view per team" — both of which add objects somebody has to keep in step.
LF-Tags invert the maintenance burden. Instead of granting a team SELECT on each of four hundred tables, you tag resources — classification=pii, domain=finance — and grant the team access to tag values. Tags inherit downward from database to table to column unless overridden, and a new table tagged domain=finance is readable by that team the moment it is tagged, with no grant written at all. Any stem combining "hundreds of tables", "dozens of teams" and "minimal ongoing administration" is asking for this.
Amazon DataZone is a layer above both. Where the Glue Data Catalog answers "what columns does this table have", DataZone answers "who owns this dataset, what does it mean, and how do I get approved to use it": producers publish assets with business metadata into a domain's inventory, consumers search and subscribe, and on approval access is provisioned through Lake Formation. It is about discovery and governed sharing between teams, and it does not replace the technical catalog underneath — a distinction a question will test by offering it as an alternative to one. Its capabilities are also surfaced through the next generation of Amazon SageMaker, so either name can appear.
Which permission model does this requirement need?
Interactive decision tree — outcomes:
- IAM and bucket policies are enough
Prefix-grain control over a small estate does not need another system. Keep it simple, and revisit when either the grain or the count changes — which is usually what triggers a Lake Formation project in the first place.
- Lake Formation, enforced
Register the locations, grant at table or column level with data filters, and — the step that makes it real — remove the direct S3 permissions so credential vending is the only path. Then check that
IAMAllowedPrincipalshas been revoked from the resources and from the database-creators list, or your grants are decorative. - Hybrid access mode first
Register the location in hybrid mode and opt principals in a few at a time. Opted-in principals need both Lake Formation and IAM permissions; everyone else carries on unaffected, so there is no flag day. Note that credential vending needs the location registered even for principals who have not opted in.
- Lake Formation with LF-Tags
Grant on tag values rather than on resources, so a table becomes readable by the right team the moment it is tagged and no grant is written per table. Tags inherit from database to table to column, which is both the convenience and the thing to review — an inherited
classification=piion a database reaches tables nobody thought about.
Key terms: AWS Lake Formation, Credential vending, IAMAllowedPrincipals, Hybrid access mode, Data filter, LF-Tag, Amazon DataZone
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.