Key takeaway: An evaluation set is the unit test suite for probabilistic systems. Without one you cannot tell improvement from regression, and every change becomes a matter of opinion.
The State Without One
A prompt gets edited. Someone tries a few examples, the output looks better, and it ships. Two weeks later a different failure is reported. Someone edits the prompt again, fixing that case and silently reintroducing the first.
This loop can continue indefinitely because nothing measures the whole. Progress feels like it is happening and the aggregate quality is a random walk.
Model upgrades make it worse. A new version is released with better benchmark scores. Whether it is better for your task is unknowable without your own measurements, and general benchmarks routinely disagree with specific applications.
Constructing One That Works
Fifty well-chosen examples outperform five hundred arbitrary ones. Composition matters more than volume.
| Category | Share | Purpose |
|---|---|---|
| Typical cases | 40% | Guards the common path |
| Known past failures | 30% | Prevents regression |
| Edge cases | 20% | Boundaries, empty input, unusual formats |
| Adversarial | 10% | Injection attempts, out-of-scope requests |
The regression category is the one that compounds in value. Every production failure becomes a permanent test case, so the same bug cannot ship twice. That single practice is most of the benefit.
Draw examples from real traffic rather than inventing them. Synthetic cases reflect what you imagined users would do, and real logs reveal what they actually do — which is consistently stranger.
Scoring Without Exact Matches
Free-text output has no single correct string, so grading needs to match the task.
Where the answer is constrained — a category, a number, an extracted field — exact match works and should be used, because it is cheap and unambiguous.
Where the output is prose, assert properties rather than text. Does it cite a source present in the retrieved context. Does it stay under the length limit. Does it avoid claims absent from the context. Does it refuse when it should. These are programmable checks.
Model-based grading handles the remainder. Give a rubric with concrete criteria rather than asking for a quality score, and validate the grader against human judgement on a sample — an uncalibrated grader produces confident numbers that measure nothing.
Running It Where It Matters
Gate merges on it. A prompt change that lowers the score should fail CI exactly as a broken test does.
Report per-category results rather than one aggregate. An overall improvement that hides a drop in adversarial handling is a security regression disguised as progress.
Keep the set stable. Changing the evaluation and the system simultaneously makes results uninterpretable. Add cases; avoid revising old ones without recording why.
The Bottom Line
Build fifty examples from real traffic before writing the feature, add every production failure permanently, score with exact match where possible and property assertions where not, and block merges on regression. Everything else in LLM development depends on having this.
