The pipeline spine, end to end

Lesson 1 of 4 in The Data Service Map.

Everything on this exam is one of seven jobs. Something has to ingest data, something has to transform it, something has to store it, something has to catalog it so it can be found, something has to serve queries against it, something has to orchestrate all of that on a schedule and retry it when it breaks, and something has to govern and observe the whole line so it is secure and provably working.

That is the spine. There are dozens of services in scope, but there are only seven jobs, and every service does exactly one of them — which is why the right first question about an unfamiliar service name is never "what does it do" but "which station is it standing at". Once you can place a service on the spine, you can reason about it even if you have never used it: a station that ingests has to deal with throughput and ordering, a station that stores has to deal with access patterns and lifecycle, a station that serves has to deal with query cost.

The board below is the one visual worth returning to on this path. Learn the shape and the rest of the course becomes a matter of filling in which service stands where.

Two properties of the spine are worth stating because the exam leans on them constantly.

The read side is a round trip, and the catalog is in it. A query engine over a lake does not read your files and work out what they mean; it asks the catalog, gets back a schema and a list of locations, and then reads. That is why a missing partition registration breaks a query that looks perfectly healthy, and why the catalog is the thing two engines share when they read the same bytes.

Governance is not a layer at the end; it is a gate in the middle. The permission check happens on the request path, which is why column-level and row-level controls are enforcement rather than documentation — and why "the data was encrypted" never answers a question about who was allowed to read it.

Key terms: Data lake, Data warehouse, Bytes scanned, Idempotency, Freshness (and the latency budget), Data lineage

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.