Crawlers, classifiers, and bad schemas

Lesson 2 of 4 in Cataloging and Discovery.

A crawler is often described as "it reads your data and works out the schema", which is close enough to be useless. What it actually does is sample, and knowing exactly how much it samples is what lets you predict when it will be wrong.

For CSV it reads the first 1,000 records or the first 1 MB, whichever comes first. For JSON it reads the first 1 MB, and if it cannot find a valid record in that it extends in 1 MB steps to a maximum of 10 MB. For Parquet it takes the schema straight from the file footer, which is why Parquet is the one format a crawler essentially cannot get wrong — the writer already declared the types.

Then it compares the schemas it inferred across every file and folder under the include path and creates one table per distinct shape. Four differences each cause a split: format, compression codec, schema, and partition structure. So a prefix containing last year's gzip CSV and this year's Snappy Parquet produces two tables, and so does one where an optional column appears in half the files.

Before the inference there is a classifier chain. Your custom classifiers run first, in the order you listed them, and the first one returning certainty 1.0 wins; if none is certain the highest-certainty result is used; only then do the built-ins for CSV, JSON, Parquet, ORC, Avro, XML and the common log formats get a turn. Writing a Grok, XML, JSON-path or CSV classifier is therefore the fix when the built-in guess is wrong — most often a headerless or unusually delimited CSV where every column lands as a string and the first data row has been eaten as column names.

A horizontal bar chart of bytes a single AWS Glue crawler reads from one 2 gigabyte file in order to infer its schema, by format. A CSV file is sampled at the first 1,000 records or the first 1 megabyte, whichever comes first, shown as about 1 megabyte. A JSON Lines file is sampled at the first 1 megabyte, shown as 1 megabyte. A JSON file in which no valid record appears in the first megabyte is extended in 1 megabyte increments to a maximum of 10 megabytes, shown as 10 megabytes. A Parquet file contributes only its footer, shown as a fraction of a megabyte, because the writer already recorded the column names and types there. The teaching point is that every inference except the Parquet one is drawn from roughly one twenty-thousandth of the file, so a column whose values change character later in the file will be typed from the beginning of it.

How much of a 2 GB file a crawler actually reads before it decides the schema. The inference is only as good as the sample, which is why Parquet is the format a crawler cannot misjudge. (illustrative — source: AWS re:Post — how does the AWS Glue crawler detect the schema)

Interactive sorting exercise: A crawler runs once over a single include path. For each situation, decide what the catalog will hold afterwards — the point being that “one table” is an outcome, not a default.

Three crawler settings that have cost people data or sleep

DELETE_FROM_DATABASE plus a briefly empty prefix. The crawler concludes the data is gone and removes the table and partition entries. The objects are untouched, but the claim about them is gone, and nothing in the catalog remembers what it used to say. DEPRECATE_IN_DATABASE gives you the same signal without the loss.

UPDATE_IN_DATABASE on a hand-tuned table. Somebody corrects a type in Athena DDL, the crawler runs that night and overwrites it with its own inference, and the error comes back. If a schema is maintained deliberately, set the policy to LOG — or stop pointing a crawler at that table at all. A crawler is for data you do not yet understand.

A crawler on a schedule over a table that only ever gains partitions. It lists the whole prefix every run to find what is new, so its runtime grows with the table. The job that wrote the data already knew which partition it created: have it call ALTER TABLE ADD PARTITION or the catalog API, and the discovery problem disappears instead of getting slower.

Key terms: AWS Glue crawler, Classifier, CombineCompatibleSchemas, Crawler schema-change policy

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.