AI & Enterprise Intelligence
AI evaluation: what the average score leaves out
An aggregate rating can hide the failures that matter most. A useful evaluation explains which tasks improved, what still goes wrong and how confidently the evidence supports a decision.
Imagine an assistant that converts maintenance notes into a draft work order. One response misses a heading. Another names the wrong machine. Both might lose points in an evaluation, yet the consequences are different. A single average obscures that difference precisely when someone has to decide whether a change is acceptable.
Evaluation becomes more useful when the report preserves those distinctions. The question is not simply whether a score rose. It is whether the assistant became better at the work, whether serious failures remain and whether the measurement itself deserves trust. That requires looking inside the result.
Write the rubric around the next decision
For the maintenance example, assess equipment identity, supporting evidence, omitted conditions and the action proposed. Keep these dimensions separate from readability. Agree what a reviewer would need to approve the draft, then describe observable differences between acceptable, incomplete and incorrect responses. Avoid a vague instruction to rate overall quality.
Use deterministic checks for properties with an explicit rule, such as a required field or an equipment identifier drawn from the supplied record. Interpretation may need expert judgment. Anthropic distinguishes code-based, model-based and human graders, noting that exact matching can reject valid variations. Choose the grader for the property being assessed. [1]
Keep a genuinely unfamiliar set of cases
Include routine requests, incomplete notes, contradictory records and cases where no supported answer exists. Review coverage by meaningful slices: different tasks, languages or source quality, where relevant. A large set of nearly identical examples tells a different story from a smaller set that exposes distinct failure modes. Record which situations are still missing.
Google's machine-learning guidance separates training, validation and test data, and warns that repeated reuse can wear out a held-out set. [2] Apply that discipline when iterating prompts, retrieval or grading rules: reserve fresh cases for a later check. Treat this as a practical engineering recommendation, and record when examples have influenced a change.
Make severity and abstention visible
A sensible report should show critical mistakes independently. Better formatting elsewhere should not cancel an incorrect machine identity. Define severity with the people responsible for the workflow: what needs a small edit, what prevents completion and what could cause a consequential wrong action? Include examples so reviewers apply the categories consistently.
Also test the decision to stop. With missing evidence, asking for the serial number may be the correct response. On a complete record, the same question could be unnecessary friction. Score appropriate clarification or abstention against the task conditions, rather than rewarding every refusal. Show how often the system completes answerable work as well as how it handles uncertainty.
Calibrate the judge against people who know the work
Google's judge-evaluation documentation uses human-rated examples to assess alignment between model-based metrics and human judgment. [3] For a business workflow, ask knowledgeable reviewers to independently rate a shared sample. Discuss disagreements before scaling the assessment. Sometimes the model is wrong; sometimes the rubric leaves a condition open to interpretation.
Retain the reviewed examples and reasons as calibration material. Check difficult cases again when the rubric or judging model changes. Agreement with a judge is not verified correctness: a plausible explanation still needs checking against the record. A numeric rating should be read as the outcome of a defined assessment, not as a probability that the answer is true.
Look for artefacts in the measurement
The 2023 MT-Bench study documented position, verbosity and self-enhancement biases, alongside reasoning limitations, in the judges it examined. These findings identify behaviours to investigate; they do not establish that every current model has identical weaknesses. [4] When comparing two responses, try reversing their order and inspect whether longer presentation is being mistaken for better substance.
Review the evaluation machinery too. A broken parser, an ambiguous expected answer or an overly strict string comparison can make valid work appear wrong. Anthropic describes grading failures that distorted results and recommends inspecting the underlying traces. [1] Keep sample outputs beside scores so a surprising change can be investigated, rather than explained away.
Decide what this particular change earns
Compare candidate and baseline on the same cases with the same rubric, and record configuration differences. Present the changed slices and serious failures before the overall mean. A model that improves difficult summaries while weakening identifier accuracy creates a tradeoff someone must explicitly accept, investigate or reject.
Keep stretch cases separate from regressions in work the system already handles. Anthropic makes this distinction between capability and regression evaluation. [1] A release discussion can then state what improved, what deteriorated, how many relevant cases were examined and where evidence remains thin. That is a more useful decision record than a leaderboard position.
Sources & further reading
The references behind this perspective. Reviewed . Product documentation can change; check the source before making an implementation decision.
- Anthropic — Demystifying evals for AI agents
Primary engineering guidance on grader selection, brittle exact matching, structured rubrics, grading failure modes and capability versus regression evaluation. Published January 2026.
- Google Developers — Datasets: Dividing the original dataset
Primary explanation of separate training, validation and test datasets and repeated-use wear-out. Applying that holdout discipline to prompt, retrieval and grader iteration is this article's engineering recommendation.
- Google Cloud — Evaluate a judge model
Primary documentation for comparing model-based evaluation metrics with a dataset containing human-rated ground truth. Supports the calibration discussion, not a claim that judge agreement verifies factual correctness.
- Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Original 2023 research documenting potential judge biases and reasoning limitations in the evaluated systems. Cited as failure modes to investigate, not as a universal claim about present-day judges.
