Lake Formation: table, column, row

Lesson 2 of 4 in Authentication and Authorization over Data.

Start with what an IAM policy genuinely cannot do, because that is the whole case for AWS Lake Formation.

An IAM policy authorises API calls on resources. For Amazon S3 the resource is an object key, so a policy can say "this role may read objects under /orders/region=EU/" — which works only because somebody chose to partition by region, and stops working the moment the question is "all regions, but not the email column". Columns are not resources. Rows are not resources. A Parquet file holds every column of every row it contains, so no S3 permission can express a column restriction, and partition-prefix tricks answer row questions only when the predicate happens to match the physical layout.

Lake Formation moves the decision up a layer. Permissions are granted on catalog objects — databases, tables, columns — and the query engine is told what it may return. Grants look like SQL (SELECT, DESCRIBE, ALTER, DROP, INSERT, CREATE_TABLE) rather than like API actions, and the principal can be an IAM role, an IAM user, or a directory user or group when Identity Center is wired in. Athena, Redshift Spectrum, Amazon EMR and AWS Glue all honour it, so one grant governs every engine instead of one bucket policy per path per team.

The mechanism that makes this precise is the data filter. A filter belongs to one table and carries two independent parts:

  • a column specification — a list of columns (and nested struct fields) to include or exclude;
  • a row filter expression — a predicate in a subset of the PartiQL WHERE grammar: =, >, <, >=, <=, <>, BETWEEN, IN, LIKE, combined with AND and OR.

How you fill those two parts is what produces the granularity, and this is worth memorising because the exam names the three levels:

  • all columns + a row expression → row-level security;
  • named columns + the all-rows wildcard (AllRowsWildcard, "Access to all rows" in the console) → column-level security;
  • named columns + a row expression → cell-level security.

You then select the filter when granting SELECT to a principal. The reader's SQL is unchanged — SELECT * FROM orders for everyone — and three readers get three different result sets.

Two things have to be true before any of that takes effect, and both are the subject of real exam questions because both fail silently.

The location must be registered. Lake Formation only vends credentials for Amazon S3 locations you have registered with it, supplying a role it can assume. An unregistered location means no credential vending, so grants on tables over that location decide nothing — the engine falls back to the caller's own IAM permissions. Opting a principal in without registering the location grants exactly nothing.

IAMAllowedPrincipals must be revoked. To avoid breaking an existing AWS Glue estate, Lake Formation grants the Super permission to the virtual IAMAllowedPrincipals group by default. While that holds, every principal an IAM policy lets through gets full access and Lake Formation permissions are not enforced on that resource at all. This is the single most common reason a carefully authored column filter has no effect, and nothing in the console flags it: the grant exists, it is simply never consulted.

The documented migration order follows from that: replicate the existing IAM permissions as Lake Formation grants, revoke Super from IAMAllowedPrincipals on each resource, register the S3 location, then clean up the old fine-grained IAM policies. Hybrid access mode exists so you can do it principal-by-principal instead of all at once: a location registered in hybrid mode enforces only CREATE_TABLE, CREATE_PARTITION and UPDATE_TABLE by default, and you opt individual principals in per database or table. An opted-in principal needs both Lake Formation permissions and IAM permissions; everyone else carries on with IAM alone. One table can therefore be a governed resource for the analytics team and a plain Glue resource for the legacy job nobody dares touch.

And the limitation that no amount of catalog configuration fixes: the direct S3 bypass. Lake Formation filters rows and columns as data leaves the query engine. It does not encrypt or restructure the files. A principal holding its own s3:GetObject on the underlying prefix downloads the Parquet file and reads every column the filter was hiding — the filter was not breached, it was never consulted. Closing that gap is a bucket-policy job: deny s3:GetObject on the data prefix to everything except the registration role and the few roles that legitimately need raw access.

Resource-by-resource grants do not scale. A few hundred tables means a few hundred grants, and every table created tomorrow needs another one — which is how a data lake ends up with a permission model nobody can describe.

LF-Tags, Lake Formation's tag-based access control, invert the relationship. You define a tag key with an allowed value list — classification = public | internal | restricted, domain = finance | hr | logistics — attach tags to databases, tables or individual columns, and grant permissions on a tag expression rather than on a resource. A role granted SELECT where classification = internal reaches every table carrying that tag, including tables created after the grant. Tags inherit downward (a database tag applies to its tables, a table tag to its columns) and a tag set at a lower level overrides the inherited one, which is how one restricted column sits inside an otherwise internal table.

The trade is diagnostic difficulty, and it is real. With resource grants, "why can this role read that column" is one grant you can read. With tags it is an evaluation over inheritance and expressions, which is why Lake Formation exposes an effective-permissions view — and why a tag taxonomy with twelve keys and free-text values is worse than no taxonomy at all. Keep the key count small and the value lists closed.

The natural division of labour: LF-Tags decide which tables, data filters decide which cells. Tags scale the coarse question across a growing catalog; filters answer the fine question within one table. A question describing "new tables must inherit the right access automatically" wants tags. A question describing "the same table, different columns per team" wants a data filter. One describing both wants both.

Which access control does this requirement need?

Interactive decision tree — outcomes:

  • LF-Tag based access control

    Define a small closed tag taxonomy, tag databases and tables, and grant on tag expressions. New tables inherit access from their tags, so the grant does not have to be reissued — which is exactly the property resource grants lack. Pair it with a Config rule or a pipeline check that refuses to publish an untagged table, because an untagged table is an ungoverned one.

  • A Lake Formation table grant

    Grant SELECT on the table to the principal and let the engine enforce it. Remember the two preconditions: the S3 location must be registered, and IAMAllowedPrincipals must have been revoked on that resource — otherwise the grant is never consulted.

  • An IAM policy and bucket policy — Lake Formation adds nothing here

    Lake Formation filters data on its way out of a query engine. An application reading objects with the S3 API never enters one, so the control has to be an IAM policy plus a bucket policy on prefixes. If a fine-grained restriction is genuinely required for that application, the answer is to put a query engine in the path, or to produce a pre-filtered dataset for it.

  • A data filter: column specification + all-rows wildcard

    Include or exclude the named columns and choose “Access to all rows”. This is column-level security: the column is absent from the result, not blanked, so a reader selecting it gets an error rather than nulls.

  • A data filter: all columns + a row filter expression

    Leave column access at “all columns” and write the predicate — region = 'EU', tenant_id IN ('a','b'). Row-level security, enforced by the engine, with the reader’s SQL untouched. Remember the expression uses a PartiQL WHERE subset, not arbitrary SQL — no subqueries.

  • A data filter with both halves populated — cell-level security

    Named columns and a row expression. This is the external-partner shape: three columns, only the rows that concern them. Create one filter per audience rather than trying to make one filter serve several.

  • Masking, not filtering — and it lives in the engine

    A data filter removes a column; it cannot return a transformed value. For “keep the column but show the last four digits”, you need Amazon Redshift dynamic data masking in the warehouse, or a masked view, or a transformation applied upstream in the pipeline. Choose by what downstream code expects: masking is transparent to a query written against the full schema, a removed column is not.

Key terms: Lake Formation data filter, LF-Tag, IAMAllowedPrincipals, Lake Formation hybrid access mode, Direct Amazon S3 bypass, Data lake administrator

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.