Defining what good looks like
Lesson 1 of 3 in Evaluating GenAI Systems.
Classic ML evaluation had a luxury you lost the day you shipped a foundation model: a single right answer. A fraud model's prediction is right or wrong, and accuracy, precision, and F1 follow mechanically. An FM answering "summarize this contract" has thousands of acceptable outputs and infinitely many subtle failures — so the exam's first evaluation skill is building an assessment framework beyond traditional ML metrics, scored along four dimensions: relevance (does it answer the question that was asked), factual accuracy (are its claims true, or supported by the provided sources), consistency (does it say the same thing when asked the same thing), and fluency (is it coherent, well-formed language).
At the Professional level you are not asked to recite these — you are asked to instrument them: which dimension a reported failure belongs to, which measurement produces a score for it, and what to wire so the score keeps being produced after launch.
| Dimension | The question it asks | How it becomes a score | The failure it catches |
|---|---|---|---|
Relevance | Did the response address this prompt, fully and without detours? | Judge model scoring against the prompt; user ratings | Beautiful answers to a question nobody asked |
Factual accuracy | Are the claims true — or, in RAG, supported by the retrieved sources? | Faithfulness and correctness metrics in RAG evaluation; grounding checks | Confident fabrication: invented specs, citations, numbers |
Consistency | Same question, same meaning in the answer — across runs, phrasings, sessions? | Repeated probes compared semantically; Semantic drift checks against a baseline | The assistant that contradicts yesterday's answer |
Fluency | Is the output coherent, well-structured language in the requested format? | Automatic metrics and judge rubrics; schema validation for structured output | Rambling, broken formatting, mid-sentence stops |
Three instruments produce those scores, and Amazon Bedrock evaluations packages all three as managed job types. Programmatic jobs compute deterministic metrics (BERTScore, F1, exact match) against reference answers — cheap and repeatable, but blind to meaning beyond overlap. LLM-as-a-judge jobs have a second model score each response against rubrics such as correctness, completeness, and harmfulness, with a written explanation per score — the scalable middle ground. Human-based jobs bring a work team (yours, or AWS-managed) where judgment outruns rubrics. A fourth job type, RAG evaluation, scores a whole retrieval pipeline in two halves — retrieval quality (context relevance, context coverage) and generation quality (correctness, completeness, Faithfulness) — so a bad answer is traced to the half that caused it.
Which evaluation mode does the scenario want?
Interactive decision tree — outcomes:
- Programmatic evaluation job
Deterministic metrics against references — BERTScore, F1, exact match — on built-in or custom prompt datasets. Cheapest and perfectly repeatable; use it where a reference answer genuinely defines success (extraction, classification-shaped tasks).
- LLM-as-a-judge evaluation job
A judge model scores each response on rubric metrics (correctness, completeness, harmfulness) and writes an explanation per score. The exam cue is the pairing of scale with per-response explanations and no human workforce.
- Human-based evaluation job
Your workforce or an AWS-managed team rates or ranks responses. Slow and costly per item — so it samples, calibrates the automated judges, and owns the calls rubrics cannot make.
- RAG evaluation job
Scores retrieval (context relevance, coverage) separately from generation (correctness, completeness, faithfulness) over a query dataset — the instrument that names the failing half of a pipeline. Works on Knowledge Bases and on custom RAG sources.
- Agent performance framework
Task completion rate, tool-usage effectiveness, and reasoning quality scored from execution traces — the third lesson builds it.
Key terms: Amazon Bedrock evaluations, LLM-as-a-judge, RAG evaluation, Faithfulness, Human evaluation (Bedrock)
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.