Evaluator–optimizer: generate, critique, refine

Lesson 5 of 6 in Workflow Patterns: Five Ways to Compose Model Calls.

Sometimes the first draft is simply not good enough, and no amount of prompt polishing fixes that in one shot. Evaluator–optimizer splits the work across two roles in a loop: a generator produces a candidate, an evaluator judges it against explicit criteria and writes actionable feedback, and the generator revises. The loop exits when the evaluator accepts — or when your code says enough.

The pattern works for the same reason human editing works: critiquing a draft is easier than writing a perfect one. A model that cannot produce flawless output in one pass can often recognize specific flaws when asked only to judge — separating the roles gives each call one job and full attention, the recurring theme of this module.

Evaluator–optimizer loop

Task arrives
Generator: produce candidate
First pass runs from the task alone; later passes also receive the evaluator’s feedback — and only the feedback, not the evaluator’s whole transcript.
Evaluator: judge against rubric
A separate call with a written rubric and a structured verdict: pass/fail per criterion plus concrete, actionable feedback. Vague rubric in, endless loop out.
Iteration budget left?
Code, not the model, owns this check. Two to three rounds capture most of the gain; beyond that you are paying full price for diminishing nits.
Accepted output
Exit with best-so-far
Ship the best candidate with the evaluator’s remaining findings attached, or escalate to a human. Never loop forever; never silently discard the work.

Failure modes: loops that never converge

The wandering evaluator. With vague criteria the evaluator finds something every round — models asked “any problems?” reliably produce problems — so the loop never terminates on quality. Convergence comes from the rubric: finite, explicit, checkable criteria, a verdict per criterion, and acceptance defined before the loop runs. If you cannot write the rubric, you do not have an evaluator–optimizer problem yet.

The oscillating loop. Round two says “too formal”, round four says “too casual” — the generator ping-pongs between contradictory critiques and the stopping condition never fires. Detect it in code: if scores stop improving across rounds, exit with best-so-far instead of burning the remaining budget.

Evaluator gaming. Iterate against the same judge long enough and the generator learns the judge’s tells — outputs that score well rather than are good, the same dynamic as reward hacking and the known weakness of LLM-as-judge. Anchor the evaluator with code-checkable criteria where possible, spot-check accepted outputs against a golden dataset, and audit samples with humans. And remember the loop multiplies cost: every round is a generate plus an evaluate — a three-round loop costs roughly six calls for one deliverable.

One boundary question completes the picture: is this still a workflow? Yes — and the distinction matters. The loop iterates, but code owns the control flow: the roles are fixed, the exit conditions are yours, and the model never chooses what happens next — it only produces drafts and verdicts inside a structure you wrote. The moment you hand the evaluator the power to decide what the generator should do next — new tools, new subtasks, new goals — you have built an agent loop wearing a workflow costume, and it needs the guardrails and budgets of one.

Key terms: evaluator–optimizer, LLM-as-judge, stopping condition, golden dataset, quality gate

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.