A user asks your assistant to summarise a contract clause. The correct answer, according to the golden reference in your eval set, is: βThe clause limits liability to direct damages only.β Your assistant returns: βUnder this clause, consequential damages are excluded; the vendorβs exposure is capped at direct losses.β Thatβs a correct, arguably better answer. Your exact-match test logs a failure.
This is the crack in the evaluation wall that opens the moment your task involves open-ended output: a model can be right in a hundred different phrasings, and exact-match can only recognise one of them.
Why this matters β and what makes it hard
For classification tasks, structured output, or fill-in-the-blank extraction, exact-match is fine. But the moment youβre grading summaries, explanations, multi-turn conversations, or any output where form and phrasing are up to the model, a string comparison becomes an unreliable judge. Two responses can be semantically equivalent yet share almost no tokens.
That gap matters in production. Misgrading correct answers as failures poisons your evaluation signal: you start tuning against the wrong target, trust scores you shouldnβt trust, and miss regressions on things that actually matter.
The alternatives
There are four common approaches before reaching for a judge model.
Manual QA is the honest baseline. Human annotators read outputs and grade them. It catches real problems and sets the quality bar. Its limits are obvious: it doesnβt scale, itβs expensive, itβs slow, and different annotators grade differently.
Per-turn scoring uses a small rubric β relevance, fluency, factual consistency β applied to each response automatically by simpler heuristics. Itβs faster than full human review but gives coarse signal; a response can score well on every rubric dimension while still being wrong about the thing that matters.
Reference metrics like BLEU were built for machine translation, where the target output space is genuinely limited. BLEU computes n-gram overlap between your modelβs output and one or more reference translations. On translation itβs correlated with quality; on open-ended text it penalises correct paraphrases and rewards fluent wrong answers. ROUGE has the same property. These metrics are fast and reproducible, but they are evaluating surface overlap, not meaning.
Public benchmarks tell you how a model performs in general, not how it performs on your task. As covered in the evaluation-set post, data contamination and task mismatch make leaderboard rankings a weak signal for production decisions.
LLM-as-judge over full transcripts, scored against a versioned golden set
The approach that scales to open-ended output is LLM-as-judge: a capable model reads your systemβs full transcript β user turn, model response, relevant context β and scores it against a rubric you define. The judge is not comparing strings; it is making a semantic judgment about quality.
The key implementation decisions are:
Judge over full transcripts, not isolated turns. A single turn can look fine and hide a problem that only appears when you read the conversation in sequence β a contradiction, a dropped constraint, a confidence that was reasonable in turn 2 and wrong by turn 5. The transcript is the unit of evaluation.
Score against a versioned golden set. A golden set is a curated collection of inputs paired with expected outputs or graded rubric scores, assembled and checked by humans. The βversionedβ part matters: whenever you update your prompt, fine-tune the model, or change the task definition, you cut a new version of the golden set. This lets you compare evaluations over time instead of wondering whether a score change reflects model behaviour or drift in the benchmark.
Gate releases in production. LLM-as-judge should sit in your deployment pipeline, not just your offline experiments. Run it before you ship a new model version and block the release if the aggregate score drops below a threshold. This makes evaluation a control, not an audit.
Zheng et al., βJudging LLM-as-a-Judge with MT-Bench and Chatbot Arenaβ (2023), established this approach empirically across the MT-Bench benchmark: GPT-4-class judges reached over 80% agreement with human raters on most categories, comparable to human-human inter-annotator agreement. They also identified and measured the biases described below β the paper is the fieldβs canonical reference for both the methodβs strengths and its failure modes.
Where it loses β the traps
Adopting LLM-as-judge without understanding its biases is replacing one broken evaluation with a different broken evaluation.
Position bias. When you present a judge model with two candidate responses to compare, it has a measurable tendency to prefer whichever response appears first β not because of content, but because of position. Zheng et al. documented this and recommended always running the evaluation in both orders and averaging the results. If youβre scoring rather than comparing, position bias manifests as the judge giving higher scores to responses earlier in the conversation.
Verbosity / length bias. Judge models tend to rate longer, more detailed responses higher, regardless of whether the extra detail is accurate or relevant. A response that adds confident-sounding elaboration to a correct core answer often scores higher than a terse, equally correct answer. This creates a perverse incentive if you use judge scores to fine-tune: the model learns to be verbose, not to be right.
Self-preference. Models from the same family or training lineage rate each otherβs outputs higher. If you use GPT-4 as your judge while evaluating GPT-4-class models, you are introducing a systematic advantage for that family. The practical fix is to use a judge from a different architecture and training lineage than the model under evaluation β or to audit the judgeβs calibration by running it against a human-rated sample.
Drift. Judge models are themselves updated. A score of 8.2 from your judge in March and a score of 8.2 in September may not mean the same thing if the judge model version changed. Treat your judge version like a library dependency: pin it, version it, and note when you update it.
The golden set is the ceiling. A judge model can only evaluate along dimensions the golden set encodes. If your golden set was built on typical cases and your failure mode is an unusual edge case β a specific document type, a language register, an adversarial input pattern β the judge will not catch it. The judge is only as trustworthy as the golden set itβs calibrated against, and it will reliably miss everything the golden set never anticipated.
The guardrails
None of these biases make LLM-as-judge unusable; they make it a tool that requires discipline.
Run your judge in both orders when comparing and average the results. Set length-normalised rubric criteria rather than open-ended quality ratings to reduce verbosity bias. Use a judge from a different model family than the model under evaluation. Pin the judge version and record it with your evaluation results. Audit a human-rated sample against judge scores periodically β a 10% random sample read by a human is the minimum. And treat the golden set as a living artefact: review it when you change the task, extend it when you find failure modes the judge missed.
The deeper discipline is recognising that evaluation is a product you maintain, not a test you run once. A versioned golden set, a pinned judge, a human audit loop, and a production gate together form an evaluation system. Informal scoring β βit looks rightβ β is not a system; itβs a guess that accumulates debt.
For the failure modes an evaluation system is designed to catch, see production failure modes. For how to build the own-set foundation that the golden set extends from, see evaluate on your own set.