How to know your LLM is right: a practical guide to AI evals

Every AI feature has two versions. There is the one in the demo, where the founder types the perfect question and the model gives a beautiful answer. And there is the one in production, where thousands of real users type things you never imagined, a prompt tweak silently breaks a case that used to work, and nobody notices until a customer complains. The thing that stands between those two versions is not a better model. It is evals — the test suite for AI.

In from prototype to production we listed evals as one of the four pillars of a trustworthy AI feature. This post zooms all the way in on that pillar: what an eval actually is, how to build your first eval set, how to score answers that have no single right form, and how to wire it all into the workflow so a regression can never ship unseen.

Why "it looks better" is not good enough

Traditional software is deterministic: the same input gives the same output, so a green test suite means the behaviour you care about still holds. AI breaks that assumption. Change a word in a prompt, swap a model version, adjust your retrieval, and the output shifts in ways you cannot fully predict. A change that fixes one case often quietly breaks three others — and because the output is fluent prose, the breakage reads fine at a glance.

Without evals you are flying on vibes: you eyeball a few answers, decide the change "feels better," and ship. Evals replace that feeling with a number. They turn "I think this prompt is better" into "this prompt scores 91% versus 84%, and here are the three cases it regressed." That is the entire game.

What an eval actually is

Strip away the tooling and an eval is three plain things:

• A dataset — a list of representative inputs, each paired with what a good outcome looks like.
• A scorer — a function that looks at the model's actual output for an input and decides how good it was.
• A threshold — the score you require before a change is allowed to ship.

That is it. An eval set is to an AI feature what a unit-test suite is to a normal codebase: a fixed collection of cases you run on every change, so you find out immediately when something you cared about stops working.

Where the cases come from

The single biggest mistake teams make is inventing eval cases at their desk. Cases you imagine are the easy ones — the happy path the demo already handles. The cases that matter come from the real world:

• Real user queries. Once you have any traffic, your logs are the best source of eval cases you will ever find. Sample real questions — especially the weird, terse and ambiguous ones.
• Every bug becomes a case. When the model gets something wrong in production, don't just patch the prompt — capture that exact input as a permanent eval case, the way you'd write a regression test for a fixed bug. It can now never silently break again.
• Known-hard inputs. Deliberately add the edge cases: empty input, a question your data can't answer, two questions at once, a prompt-injection attempt, another language.

You do not need thousands to start. Fifty carefully chosen cases that cover your real distribution — the common paths, the tricky ones, and the "should refuse" ones — will catch more regressions than a vague sense that things seem okay. Start there and grow the set every time production surprises you.

Scoring: the genuinely hard part

Deciding whether an answer is "good" is where evals get interesting, because most AI outputs have no single correct string. There is a ladder of scoring methods, and you climb it only as far as the task forces you to:

• Exact / structured match. When the output should be a label, a number, a SQL query or JSON, you can check it directly — did it classify correctly, does the JSON parse, does the query return the right rows. Cheap, deterministic, unambiguous. Use it wherever you can.
• Rule-based checks. For freer text you can still assert facts: does the answer contain the required figure, does it stay under the length limit, does it cite a source, does it avoid a banned claim. A stack of small boolean checks goes a surprisingly long way.
• LLM-as-judge. For open-ended answers — a support reply, a summary, a rewrite — you hand the input, the model's answer and a rubric to a second model and ask it to grade against that rubric. It is the only method that scales to subjective quality, and it is the one to treat with the most suspicion.

Using a model to grade a model, safely

LLM-as-judge is powerful and treacherous in equal measure. A vague instruction like "rate this answer 1–10" produces noise — the same answer scores 6 one run and 8 the next. The fixes that make it reliable:

• Give it a rubric, not a vibe. Ask specific yes/no questions — "Is every factual claim supported by the provided context?", "Does it answer the question that was asked?" — instead of a single fuzzy score.
• Prefer binary or low-cardinality verdicts. Pass/fail, or a 3-point scale, is far more stable than 1–10.
• Show it the expected answer when you have one, so it grades against a reference rather than its own opinion.
• Calibrate the judge itself. Hand-label a few dozen cases yourself, then check that the judge agrees with you. If it doesn't, your scores are fiction — fix the rubric before you trust a single number.

What it looks like in code

The mechanics are humble on purpose — a loop over cases, a scorer, an aggregate, a threshold:

# an eval run, in spirit
cases = load_cases("evals/support.jsonl")   # {input, expected, checks}
results = []
for case in cases:
    output = feature(case["input"])        # the thing under test
    score  = judge(case, output)             # exact / rules / LLM-as-judge
    results.append(score)

passed = sum(results) / len(results)
print(f"score: {passed:.0%} on {len(cases)} cases")
if passed < THRESHOLD:                      # the gate that stops a regression
    fail("below bar — do not ship")

Notice what the last two lines do: they turn the eval from a report you might read into a gate that blocks a bad change automatically. That is the difference between having evals and being protected by them.

Run them where they'll actually stop a regression

An eval set that lives on someone's laptop and gets run "when we remember" protects nobody. The payoff comes from making evals routine and unavoidable:

• On every change. Run the suite in CI on any pull request that touches a prompt, a model version or the retrieval pipeline — the same reflex as running unit tests.
• Compare, don't just measure. Report the new score against the current baseline and list exactly which cases flipped from pass to fail. A single regressed case is often more informative than the headline average.
• Keep evaluating in production. Sample live traffic, score it with the same judges, and watch for drift — the model provider ships a new version, your users' questions shift, and yesterday's 92% quietly becomes 85%.

Common ways evals go wrong

• The set only has easy cases. If everything passes at 100%, your evals aren't testing anything — they're missing the hard inputs where the feature actually fails.
• An unvalidated judge. Trusting an LLM grader you never checked against your own labels means optimising toward a number that may not mean what you think.
• Chasing the average. A rise from 88% to 90% can hide a critical case — "refuse to give medical advice" — flipping to fail. Weight the cases that carry real risk.
• Evals that never grow. A static set slowly stops reflecting reality. Every production miss should become a new case; the suite should get harder over time.

Evals are the least glamorous part of building with AI and the one that most separates a feature people trust from a demo they abandon. They are also what make everything else safe to move fast on — you can swap models, rewrite prompts and refactor retrieval with confidence, because the suite tells you the instant you break something. It is the same discipline behind the knowledge assistant that knows when to stay quiet and the model choices we make per task.

Building an AI feature and not sure how you'll know it stays correct once it's live? That question — how do we measure this, and how do we keep it honest in production — is exactly where we like to start.

Talk to us about your AI project