The traps that sink strong engineers

Before you commit: the four checks

Interactive decision tree — outcomes:

• Commit, and move on

  All four checks passed. Record the answer, and flag it only if you want a second look — never instead of answering. The whole routine should take about ten seconds once it is a habit, which is why it is worth drilling now rather than inventing it in the room.

• You answered the un-negated question

  This is the most expensive reading error on the exam, and the cheapest to fix. Re-read the question sentence, then evaluate each option against the words actually printed: which of these does not reduce cost has a correct answer that would be wrong under the other wording. Expect roughly one or two negated items per sitting, and expect them to look easy.

• You have over-engineered it

  A queue nobody asked for, a replica no clause requires, a validation stage the stem did not mention: at work these are good instincts. In an exam item they are the signature of a distractor, because every one of them adds cost, latency or operational effort that the stem gave you no reason to accept. Prefer the simplest option that satisfies every stated clause — and then re-read the clauses to be sure it does.

• You optimised the wrong thing

  Your option is better — on an axis the question did not name. The classic pair: the stem asks for the least operational overhead and you chose the cheaper option, or it asks for the most cost-effective and you chose the one with less to manage. These genuinely diverge: a continuously running cluster at high utilisation can be cheaper and more work. Go back to the question sentence and pick on its word.

• Check the copies before you commit

  Governance clauses are usually breached by a second copy rather than the primary path: a backup with a default cross-Region destination, a read replica in another Region, a replication rule somebody set up for durability, a service-managed snapshot. Residency, retention and "who may see this" all have to hold for every copy, so re-read your option asking specifically: where else does this data come to rest, and who can read it there?

Trap: monitoring is not audit, and neither is configuration history

Three services answer three adjacent questions, and the one you reach for first is usually the one you use most at work.

• "Is it working? Did the run fail? What was the latency?" — metrics, logs and alarms. This is operational, for you, now.
• "Who called which API, and when? Who read that object?" — the API call trail, with data events enabled for object-level reads. This is evidence, for somebody else, later.
• "What configuration did this bucket have on 3 March?" — configuration history. A call record shows the moments of change; only a configuration record can tell you the state between them.

The tell in a stem is the verb. Alert, detect and investigate now point to monitoring. Prove, demonstrate to an auditor and for the last three years point to the audit trail. What was it set to at the time points to configuration history.

Trap: encryption is not authorization

"The data is encrypted at rest with a customer managed key" is a true and comforting sentence that answers no question about who may read the data. Any principal authorised for the table and permitted to use the key reads plaintext, because the service decrypts transparently on their behalf.

So when a stem says one team must not see two columns, the answer lives in the permission layer — table, column and row grants — not in the key. Encryption answers a different threat: somebody who obtains the stored bytes without going through the service.

The one place the key is the control is cross-account and cross-boundary access, where denying use of the key denies decryption regardless of what the data-layer permissions say. That is a narrower claim than "encryption controls access", and it is the version worth carrying.

Trap: the slow query and the bigger engine

When a query gets slower with no change to its SQL, the cause is nearly always in the data rather than in the engine. Three candidates, in order of how often they are the answer:

1. File layout. The table is row-oriented, or unpartitioned, or partitioned on a column nobody filters on, so every query reads far more than it needs.
2. Small files. A short delivery buffer or a chatty job has produced hundreds of thousands of tiny objects, and the per-file overhead now dominates.
3. Data growth past a design assumption. A partition scheme that was fine at a gigabyte a day is not fine at a terabyte, and a scan that was acceptable has become the whole cost.

Adding capacity makes all three cheaper to endure and none of them better. The option that adds compute is a distractor; the option that changes the layout is the fix.

Trap: “most cost-effective” is not “cheapest service”

Cost-effectiveness is cost for the stated usage, which is why the usage pattern in the stem is load-bearing rather than scenery.

Eight minutes of work once a day favours per-run serverless capacity so strongly that nothing else is close. Twenty hours a day at near-constant utilisation favours a long-running cluster with reserved or Spot capacity, and the serverless option becomes the expensive one. The same two services swap places on the strength of one sentence about how often the thing runs.

The other half of the trap is scope: the cheapest component is rarely the cheapest solution. A colder storage class with retrieval charges, a smaller cluster that runs four times as long, a cheaper store that forces a second copy for reporting — each wins on its own line and loses on the total.

Trap: the “first action” that is really the third

When a stem asks what to do first, the three wrong answers are usually all genuinely part of the fix. That is what makes it hard: nothing is incorrect as engineering, so you cannot eliminate on correctness.

The ordering that holds almost everywhere in data engineering is: stop it getting worse → preserve what you need to diagnose → establish the facts → repair → reprocess → prevent. Disabling the schedule comes before fixing the logic; fixing the logic comes before the backfill; cleaning the output table comes after the thing writing to it has stopped.

If two options are both plausibly first, ask which one the other depends on. The one with no prerequisite is first.

Trap: the catalog you forgot to write to

An incremental job writes today's files to the right prefix and the query returns nothing new. Everything about the job looks healthy, the files are visibly there, and the failure is in metadata: the partition was never registered, so the engine was never told to open that path.

Three ways this is resolved, and a stem usually hints at which one it wants: the job registers partitions as it writes; a crawler runs afterwards and discovers them; or the table is defined so that partitions are projected from the path pattern, with no registration step at all. The third is the one candidates forget, and it is the right answer when a stem complains about the time a crawler takes on a table with a very large number of partitions.