Evaluating GenAI: BLEU, ROUGE, BERTScore, LLM-as-a-judge
Lesson 3 of 4 in Evaluating Models — Classic and GenAI.
Generated text has no single right answer, so GenAI evaluation stacks imperfect lenses. N-gram overlap: BLEU (precision-flavored, born for machine translation) and ROUGE (recall-flavored, standard for summarization) count matching word sequences against a reference — cheap, reproducible, and blind to paraphrase. Embedding similarity: BERTScore and semantic-similarity metrics compare meanings, forgiving different wording — but a fluent, on-topic hallucination still scores well. LLM-as-a-judge: a strong model grades outputs against a rubric — scalable and closest to human judgment, with biases of its own. Humans: the gold standard, slow and expensive, reserved for what matters most.
| Method | Measures | Misses | Cost |
|---|---|---|---|
BLEU | N-gram precision vs reference (translation) | Paraphrase, meaning, fluency | Trivial |
ROUGE | N-gram recall vs reference (summarization) | Whether the "covered" content is actually faithful | Trivial |
BERTScore / semantic similarity | Embedding-level meaning overlap | Factuality — confident nonsense embeds beautifully | Low |
LLM-as-a-judge | Rubric-based quality at scale (helpfulness, faithfulness…) | Its own biases: position, verbosity, self-preference | Moderate (judge tokens) |
Human review | Whatever you train reviewers to see — the reference standard | Scale; consistency across reviewers | High |
On AWS, this stack is productized. Amazon Bedrock evaluations run automatic evaluation jobs (programmatic metrics over your dataset), LLM-as-a-judge jobs, and human-based evaluations with your own work teams — model-vs-model comparisons included. Bedrock Prompt Management versions prompts so experiments compare like with like: a prompt change is a model change and deserves the same rigor. For RAG systems, evaluate the retriever on its own: monitor retrieval relevance — did the retrieved chunks actually contain the answer? — separately from generation quality, or you will tune the wrong component.
Interactive sorting exercise: Match each evaluation need to the method that actually measures it.
Key terms: LLM-as-a-judge, Retrieval-augmented generation (RAG), Managed MLflow, F1 score
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.