Arenas and Ratings
Lesson 2 of 3 in Human Eval, Arenas, and LLM Judges.
A proper human round is expensive, so an obvious question follows: what if the users themselves did the rating, for free, at internet scale? That is an Arena. The mechanic: a visitor types any prompt they like; two anonymous models answer side by side; the visitor votes for the better answer (or a tie); only after voting are the identities revealed. Each vote is one blind pairwise comparison — the same head-to-head judgment your raters made in lesson one, minus the rubric, the training, and the agreement checks, plus enormous volume.
Single votes mean little; the aggregate is the product. Millions of votes across many model pairs are fitted with a rating model from the chess-rating family — Elo and its relatives, Bradley-Terry-style models that assign each model one strength score such that the predicted win probability between any two models (a function of their score gap) matches the observed votes as closely as possible. The result is the familiar leaderboard: a single ranked list distilled from pairwise preferences. Chatbot Arena (Chiang et al., 2024) is the best-documented example of the genre, and the source for most of what is publicly known about how these systems behave.
A three-by-three heatmap with models A, B, and C as both rows and columns. Cells show toy pairwise win rates: model A beats B in 65% of votes and C in 80%; model B beats C in 70%. The diagonal is 50%. Darker cells indicate higher win rates, and the pattern implies the ranking A above B above C.
Arenas fix real weaknesses of static benchmarks. There is no fixed test set to leak into training data, so Contamination in the usual sense does not apply. The prompts are live and endlessly varied rather than frozen in 2020. And the metric — blind human preference — is much closer to “which assistant would people rather use” than any multiple-choice accuracy number.
But be precise about what the number is: aggregate crowd preference, on prompts the crowd chose to ask, judged by whoever showed up to vote. Every phrase carries a limitation. The prompt distribution is whatever arena visitors type — heavy on whatever draws people to a model-comparison site — not your support tickets, your legal drafts, your codebase. The voters are self-selected and apply no shared rubric; a vote rewards whatever impressed that person in that moment. And known dynamics blur the signal further: votes appear to reward style as well as substance — longer, confidently structured, well-formatted answers tend to attract preference somewhat independently of correctness — and the prompt mix shifts as the visitor population changes, so ratings can move without any model changing. Arena operators study and partially adjust for such effects (style-controlled views of rankings exist for exactly this reason), but no adjustment makes a crowd’s preference equal your task’s quality bar.
Why must the models be anonymous and the sides randomized?
Brand names carry expectations — a vote cast knowing “this one is from the famous lab” measures reputation, not the answer. Blinding removes identity effects. Randomizing which side each model appears on removes position effects: humans, like the LLM judges of the next lesson, are not neutral about presentation order, and over millions of votes even a small side preference would systematically favor whichever model landed there more often.
What is the rating model actually doing?
Qualitatively: it assumes each model has a single latent strength, and that the probability of winning a vote depends on the gap between two strengths. It then finds the set of strengths that best explains all observed votes at once. That is the Bradley-Terry idea, and the chess Elo system is the same family. The payoff over raw win percentages is comparability: a model is not punished for having faced stronger opponents, because every vote is interpreted relative to who it was against.
Can a model’s rank move while its weights never change?
Yes, in several documented ways: new models enter and old ones leave, changing the opponent pool; the visitor population and its prompt mix shift over time (Drift in the eval itself, not the model); and tied or low-information votes accumulate differently across pairs. A leaderboard is a living estimate under a moving distribution — read trends, not decimal places.
Is a vote the same thing as the preference data used in training?
Mechanically it is the same object — a pairwise human preference between two completions — which is why it may look familiar: Reinforcement learning from human feedback (RLHF) trains reward models on exactly such comparisons. The difference is purpose and hygiene: arena votes evaluate finished models on crowd prompts, while preference datasets for training are curated, guideline-driven, and aimed at teaching a model what to prefer.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.