Human Evaluation

Lesson 1 of 3 in Human Eval, Arenas, and LLM Judges.

The previous module dealt with benchmarks — tasks with an answer key, where a script can mark each output right or wrong. Most of what you actually ship has no answer key. “Is this summary faithful to the document?” “Is this refusal appropriate or overcautious?” “Does this reply sound like our brand?” These are judgment calls, and for judgment calls, trained human judgment is the reference standard. Every other method in this module — arena votes, LLM judges — ultimately borrows its authority from agreement with humans.

That authority is easy to claim and hard to earn. The naive version — paste two outputs into a channel and ask a coworker which looks better — produces a number, but not a measurement. Different people silently score different things: one rewards thoroughness, another concision, a third formatting. Ask them again on Friday afternoon and the answers change. What separates measurement from vibes is process: a written definition of “good”, raters trained on that definition, and proof that independent raters applying it reach the same verdicts.

A human-evaluation round that earns its numbers

Define what “good” means
Turn the vague goal into named criteria with a Rubric: described score levels per criterion, not a single holistic “rate 1–10”.
Write annotation guidelines
Worked examples for each score level, plus explicit rulings on the edge cases raters will actually hit.
Pilot on a small batch
Several raters grade the same items independently, without discussing them.
Do raters agree enough?
Compare verdicts on the shared items. Frequent disagreement means the rubric or guidelines are ambiguous — not that the raters are careless.
Revise rubric and guidelines
Every disagreement is a hole in the guidelines. Add the ruling, then pilot again.
Run the full round
Blind raters to which model produced each output, randomize item order per rater, and keep a slice of items graded by multiple raters.
Aggregate and audit
Check agreement on the overlapping slice, and watch recurring anchor items for drift across sessions.
Scores with known reliability
The result is not just a score — it is a score plus evidence that the score is reproducible.

The load-bearing concept in that flow is inter-rater agreement. Give the same item to two trained raters, independently. If they usually reach the same verdict, the rubric is doing its job and the score means something. If they frequently disagree, the “score” is mostly a measurement of who happened to grade each item — and any model comparison built on it is noise wearing a number. Statisticians have chance-corrected agreement coefficients for quantifying this; the concept matters more than any formula: measure agreement before you trust the scores, and treat low agreement as a defect in the guidelines, not in the people.

The second enemy is time. Raters are human: they tire within a session, and their internal standard slides across days — the two hundredth mediocre summary looks better than the tenth did, or a rater who just read three terrible outputs scores an average one generously. This is rater fatigue and Drift, and well-run programs design against both: short grading sessions, randomized item order (so late-session leniency does not systematically favor one model), overlap between raters throughout the round rather than only in the pilot, and recurring anchor items — the same items quietly re-inserted over time, so a shift in their scores exposes a shift in the raters.

All of this is why human evaluation is expensive. It is not the minutes per item; it is the machinery around the minutes — guideline writing, pilots, overlap, audits. That expense is also exactly what the rest of this module is about: arenas crowdsource the judgment, and LLM judges automate it. Both inherit their meaning from how well they track what a properly run human round would say.

The evaluation methods of this domain, compared qualitatively. Each trades away some of human judgment’s fidelity to buy scale — the craft is knowing which trade your question tolerates.
Method	Cost per item	Reliability	Scale	Best for
Answer-keyed benchmark	Near zero once built	High on what the key can check — exact answers, passing tests	Unlimited reruns	Capabilities with checkable answers; Regression testing on every change
Trained human raters	Highest — rater time plus guideline upkeep	The reference standard, when run with guidelines, blinding, and agreement checks	Hundreds to low thousands of items per round	Safety-adjacent calls, subjective quality, and calibrating every other method
Crowd arena votes	Carried by the voting crowd	Meaningful in aggregate — about crowd preference, not your rubric	Very large, continuous	Ranking general-purpose assistants against each other (next lesson)
LLM-as-judge	Cheap — one model call per verdict	Useful once calibrated against humans; systematically biased out of the box	Near unlimited, minutes not weeks	Fast iteration between human rounds (lesson three)

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.