The Glue Data Catalog

Lesson 1 of 4 in Cataloging and Discovery.

Files in Amazon S3 are not a table. A table is a claim about files — these objects, under this prefix, hold these named and typed columns, serialised this way, partitioned by these keys — and that claim has to live somewhere outside the files for any engine to act on it.

The AWS Glue Data Catalog is where it lives: one account-wide, Region-wide metastore holding databases, tables, columns, partitions, connections and schemas. It is the managed implementation of the Hive metastore model, which is why the same table definition is read by every engine at once — Amazon Athena, Amazon EMR, Amazon Redshift Spectrum, AWS Glue ETL, Amazon SageMaker and Apache Spark all speak it. Register a table once and all of them can see it; break the definition once and all of them break.

Two things about it decide most questions.

It holds metadata, never data. A DROP TABLE on an external table removes the claim and leaves every byte in S3 untouched. That is a feature — the same files can carry two different table definitions for two different purposes — and it is the reason a lake can lose track of itself, because nothing forces the claim and the files to agree.

Its permissions are a separate decision from the data's. A principal needs catalog permissions to see that a table exists and S3 permissions to read its bytes, and those are two independent grants in the default model. Being able to list a table you cannot read is normal; the converse — reading a prefix whose table you were never granted — is the hole that AWS Lake Formation exists to close.

Five ways a table gets into the catalog, and what each one costs you
How	Good at	Bad at	Use it when
A crawler	discovering a schema and partitions you do not already know	preserving a schema you tuned by hand — it will overwrite it on the next run	onboarding unfamiliar data, or a prefix whose partitions appear unpredictably
The job that writes the data	staying in step by construction — the writer declares what it wrote	nothing, which is why this is the default for pipelines you own	you control the producer; add the partition in the same job that created it
DDL in Athena	an exact, reviewable definition with chosen types and SerDe properties	keeping up with new partitions, which it does not do at all	the schema matters more than the convenience — including when a crawler guessed wrong
Infrastructure as code	reproducibility and review: the table definition is in version control	schemas that genuinely change often, where every change is a deployment	the table is part of a product rather than an exploration
A migration tool	importing an existing Hive metastore or a source database’s structure wholesale	the inevitable fidelity gaps between one type system and another	a one-off migration, followed by one of the options above

Two practical properties of the catalog turn up in real incidents, so they turn up in questions.

Catalog calls are API calls, and they are on the critical path of every query. An engine asks GetTable once and then GetPartitions — paginated — before it reads any data. For a table with a handful of partitions that is noise. For a table with millions it is the dominant cost of the query, and it is why the next two lessons exist: partition count is a design variable, not an accident, and the catalog has an index to make the call selective.

A table can be larger than the thing describing it comfortably handles. The catalog has service quotas — partitions per table, tables per database, and so on — and a table partitioned by hour and customer will reach them. When you hear a stem say "the table has grown to millions of partitions", the correct instinct is not a bigger quota; it is a coarser partition grain, a partition index, partition projection, or a table format that prunes on file statistics rather than on directory structure.

And one habit worth forming: treat a table definition as an interface. Four engines and an unknown number of downstream consumers resolve it by name. A type change that one team makes to unblock a job is a change to everybody's contract, which is why the useful review question is not "does this work" but "who else reads this".

When the catalog and the files disagree

You will meet three errors in this family, and they are diagnosed by the same question: what does the claim say, and what is actually in the file?

HIVE_BAD_DATA — type mismatch. The catalog declares int, the file holds "7" or 7.4 or null as a literal string. Fix the producer, or widen the column type and accept the cast; do not "fix" it by declaring everything a string, which throws away every predicate pushdown the format offered.

HIVE_PARTITION_SCHEMA_MISMATCH — the partition disagrees with the table. Partitions carry their own stored schema, so a table altered after some partitions were registered has partitions describing the old shape. Re-register the affected partitions; the table-level change did not reach them.

Zero rows from a table that obviously has data. Almost always the partitions were never registered — the files are there and the claim says nothing about them. Check whether anything actually ran ALTER TABLE ADD PARTITION, a crawler, or MSCK REPAIR TABLE after the write.

Key terms: Glue Data Catalog, Hive metastore, Schema-on-read, Data lake

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.