When worker three fails

Lesson 4 of 5 in Orchestration End to End: One Request Through the Whole System.

Warehouse 2 sits behind an ERP that answers in 200 milliseconds or in 30 seconds, with no pattern anyone has found. Your fan-out has three branches. Two return; one is still open when the deadline arrives.

This is not the tail. This is Tuesday. And in most implementations it is the case with no code written for it, because fan-out is usually built and tested with all branches succeeding. There are three outcomes to design for — all good, some good, none good — and the middle one is where systems are missing.

The partial-result decision, made explicitly

Branches dispatched with a shared deadline
Deadline reached — 2 of 3 returned
The moment that needs a policy. Waiting longer, proceeding, and giving up are all defensible; choosing by accident is not.
Is the missing branch load-bearing?
Declared per branch at design time: required, or best-effort. Warehouse stock is best-effort (two suffice); the entitlement clause is required (there is no remedy without it).
Proceed, labelled partial
The merged result carries coverage: 2/3 and a note naming what is missing. The proposal downstream can promise availability only from what was actually checked.
Retry inside the remaining budget
Once, with jitter, only if the remaining deadline exceeds the observed p95 for that call. A retry that cannot finish in time is pure cost.
Degrade to a human
Not a failure state — a designed outcome. The human receives what was gathered plus what is missing, and finishes the job.
Merge with explicit precedence
Entitlement constrains which remedies are legal; availability constrains what may be promised. Code applies the rule; conflicts it cannot resolve go to the human.
Continue to propose

A merge step that concatenates is not a merge step

The most common fan-in implementation joins the branch outputs into one string and hands it to the model with “here are the results.” This works in the demo and fails in three specific ways.

It launders missing data as absence of a problem. Two warehouse reports with no mention of the third reads, to the model, exactly like “there are two warehouses”. Nothing lies; the model just has no way to know something is missing. Coverage must be a field, not an inference.

It hides disagreement. The entitlement says credit-only; the customer needs units by Friday. Concatenated, these become two sentences the model resolves by picking a side — fluently, and differently on different runs. Precedence between constraints is a business rule, and business rules belong in code.

It buries the deadline. “Warehouse 3 will have stock in 9 days” inside a paragraph is a fact the model may or may not weigh against Friday. As a structured field compared in code, it is a decision.

The general form: the merge is where partial, conflicting, and stale results become one answer, and each of those three transformations needs a stated rule. Write the rule down; then a disagreement your rule cannot settle escalates instead of resolving at random.

Partial failure gets substantially harder once a hop has already changed the world. The execution step may reship and issue a partial credit. Suppose the reship succeeds and the credit call fails.

You cannot retry the run — that would ship twice. You cannot abandon it — the customer has a pallet on the way and no credit. This is the case that separates systems that merely work from systems you can run.

Idempotency: make the retry harmless

Every side-effecting call carries a key derived from run identity, not from a timestamp or a random value: {run_id}:{step_id}:reship. The downstream system treats a repeat of that key as the same request and returns the original result.

The subtlety that bites: the key must be stable across retries and distinct across legitimate repeats. If the customer reports a second shortage on the same order, that is a new run and therefore a new key. If your key is derived from {order_id}:reship, the second genuine reship is silently swallowed as a duplicate — a failure mode that looks like the system working.

Compensating actions: define the undo before you write the do

Some effects cannot be prevented, only reversed. For each side-effecting tool, write down its compensating action at design time: cancel the shipment while it is still in the warehouse, reverse the credit memo, void the purchase order.

Two honest constraints. Compensation is often partial — a shipment already on a truck cannot be un-shipped, only recalled at a cost. And compensation can itself fail, which is what a dead-letter queue plus a human owner is for. The design goal is not perfect rollback; it is that no failure leaves the world in a state nobody can read.

Ordering: put the reversible effects first

If a step performs several effects, order them so that the hardest to reverse happens last. Issue the credit memo (reversible with an accounting entry) before dispatching the pallet (reversible only with a truck). Then a mid-step failure leaves you needing the cheap compensation, not the expensive one.

This costs nothing to design and is nearly impossible to retrofit, because by then the ordering is load-bearing for something else.

What the human gate should show after a partial failure

The state of the world, not the state of the run. “Reship 4 × SKU-4471 dispatched (shipment SH-77120); credit of $412 failed — accounting API returned 503; compensating action available: cancel shipment SH-77120 (recallable until 18:00).”

Compare with what teams usually surface: “Run failed at step 7.” The first lets a human finish the job in one minute. The second guarantees somebody opens four consoles at 2 a.m. — and the difference is entirely in what you chose to record at the seam.

Key terms: idempotency, compensating action, dead-letter queue, graceful degradation, retry budget

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.