Evaluation · September 9, 2026
Testing LLM Applications Without Fooling Yourself
Most of an LLM app is ordinary deterministic code and should be tested as such. The model-dependent part needs evals, and evals have failure modes that produce confident, wrong numbers.
By Shihab Shahriar Antor · Updated 2026-09-09
The reason testing an LLM application feels impossible is usually a category error. People look at the system, see that the output is non-deterministic, and conclude the whole thing resists testing. Then they either write nothing, or they write an eval suite that calls a model on every commit and costs more than it catches.
In practice most of the code is deterministic. The model-dependent surface is smaller than it looks, and once it is isolated the two halves get completely different treatment.
What is actually deterministic
| Component | Deterministic? | How to test it |
|---|---|---|
| Prompt template rendering | Yes | Snapshot the rendered string. Catches a variable that silently interpolates as undefined, which is the single most common real bug and produces plausible output rather than an error. |
| Tool and function schemas | Yes | Validate against the provider's schema rules in CI. Invalid schemas fail at call time in production and pass every local test that never calls the API. |
| Output parsing and validation | Yes | Table-driven tests over recorded model outputs, including the malformed ones. Keep every parse failure you have seen as a fixture, permanently. |
| Retry, fallback, circuit breaking | Yes | Fake the provider. Assert on the sequence of calls, not on content. This is where the expensive production incidents live and it needs no model at all. |
| Retrieval ranking | Yes, given fixed corpus and query | Score recall and nDCG against labelled relevance. Mechanical and cheap; no judge model required. |
| The model's answer | No | Eval suite. Everything difficult is here, and it is a much smaller surface than the list above. |
Roughly speaking, if the same input produces the same output on a machine with no network, it belongs in the unit test suite and running it should cost nothing.
Three kinds of eval, used for different things
Once the deterministic half is covered, what remains is judging output quality. There are three mechanisms and they are not interchangeable.
| Mechanism | How it works | Use it for | Where it misleads |
|---|---|---|---|
| Assertion | Deterministic checks on the output: valid JSON, contains a required field, cites a document that exists, stays under a length, never emits a forbidden string. | Contracts and safety rails. Cheap enough to run on every case, every time. | Says nothing about whether the answer is any good. Passing all assertions is a floor, not a score. |
| Reference comparison | Compare against a known-correct answer, by exact match, F1 over spans, or embedding similarity. | Tasks with a real ground truth: extraction, classification, retrieval, structured parsing. | Breaks on open-ended generation. Surface-form metrics punish a correct answer worded differently, and embedding similarity rewards being on-topic rather than being right. |
| Model as judge | A second model grades the output against a written rubric, absolute or pairwise. | Open-ended quality where no single reference exists. | The judge has biases with known direction. Treat its score as an instrument reading with error bars, never as ground truth. |
The judge biases worth knowing before you trust a number
Model-as-judge is the mechanism most teams end up leaning on, and it fails in patterned ways rather than randomly. Patterned error is correctable; random error is not, so this is good news as long as you know the patterns.
- 01
Position bias
In pairwise comparison the judge favours one slot, often the first, at rates far above chance. Mitigation is trivial: run every pair twice with the order swapped and count a disagreement as a tie. If you are not swapping, your win rates are partly measuring slot order.
- 02
Self-preference
Judges tend to score outputs from their own family higher. If the system under test and the judge are the same model, the score is inflated by an unknown amount. Use a different family as judge, and say in the writeup which one.
- 03
Length bias
Longer, more confident-sounding answers score higher independent of correctness. Watch the correlation between output length and score across your set. If it is strong, you are partly measuring verbosity and your rubric needs a brevity clause.
- 04
Rubric drift
A vague rubric produces scores that move when nothing changed. Rubrics should be concrete enough that two people applying them to the same output agree. Check that occasionally by actually doing it.
- 05
Single-judge variance
Judge choice moves aggregate scores by several points. One judge gives a number with no error bar. Two judges from different families bound the uncertainty, and the gap between them is itself informative.
What to run when
Eval suites cost money and minutes, so running everything on every commit is how a suite ends up disabled. Tier it.
| Trigger | What runs | Budget |
|---|---|---|
| Every commit | Unit tests over the deterministic half, plus assertion evals against recorded fixtures. No live model calls. | Seconds, zero cost. If this tier calls a provider, the split in the first table was not done properly. |
| Every pull request touching prompts, tools or retrieval | A stratified subset of the eval set, sized to run in a few minutes, with per-category results rather than one number. | Minutes, small and predictable. |
| Nightly and before release | The full set, both judges, latency and cost recorded per case. | Whatever it takes. This is the number you report. |
| Continuously in production | Sample real traffic, run assertion evals online, log traces. Route user thumbs-down into the eval set. | A small percentage of live volume. |
The last row is the one that compounds. An eval set built from real failures gets more predictive every month, while a set written up front stays exactly as representative as your guesses were on day one.
Report per category, never as one number
A single aggregate score hides everything you would act on. A suite that goes from 71 to 73 tells you nothing; the same run broken out by category routinely shows one category up eight points and another down five, which is a regression the aggregate concealed.
The same applies to the retrieval and generation split in a RAG system. Recall says whether the right document was found. Groundedness says whether the model then used it. They fail independently, and collapsing them means every investigation starts by re-deriving which half broke.
Tooling
The category is well served and the tools are not that different from each other. promptfoo is config-driven and the fastest way to get assertion evals into CI. DeepEval reads as a unit test framework, which fits teams that want evals in the existing test runner. Inspect is built for rigorous model evaluation and is the most serious option if the evaluation itself is the deliverable. Ragas is RAG-specific and worth it when retrieval quality is the question.
For tracing, the OpenTelemetry GenAI semantic conventions are worth adopting early. Vendor-specific tracing is easy to add and expensive to leave, and this is one of the few places where the standard arrived before the lock-in.
The uncomfortable part
An eval score is a measurement of your eval set, not of your system. If the set is unrepresentative, a rising score means the system is getting better at your set. That failure is invisible from inside the numbers, which is why sampling real traffic into the set matters more than any choice of framework or metric.
We wrote up how this plays out when the thing being measured is a memory system, including where our own numbers are not trustworthy, in why agent memory benchmarks cannot be trusted.
Questions
- How do you test a non-deterministic LLM application?
- By testing the deterministic parts as ordinary code and the model-dependent part as an eval. Prompt rendering, tool schemas, parsing, retries and routing are all deterministic and belong in a normal unit test suite. Only the quality of the model's output needs an eval suite, and that suite is scored statistically over a set rather than asserted case by case.
- What is the difference between testing and evaluation for LLM apps?
- A test has a binary pass or fail on a single input and should be deterministic. An eval measures quality across a dataset and produces a distribution, so a single case failing is expected and meaningless on its own. Tests gate every commit. Evals gate releases and get compared between versions.
- Is LLM-as-a-judge reliable?
- Reliable enough to be useful, with known biases you have to correct for. Judges show position bias in pairwise comparison, a preference for outputs from their own model family, and a preference for longer answers. Swap the order on every pair, use a judge from a different family than the system under test, and use two judges so the score has an error bar.
- Does setting temperature to 0 make LLM output deterministic?
- No. It makes token selection greedy, but the underlying computation can still vary because floating point reduction order depends on batch composition, mixture-of-experts routing can be batch-sensitive, and providers change serving infrastructure without changing the model name. Do not write tests that assert exact output strings from a live model.
- How large should an LLM eval set be?
- Large enough that the difference you care about is bigger than the noise, which usually means a few hundred cases per category rather than thousands overall. Coverage of distinct categories matters far more than raw count, since 500 cases spread over eight failure modes is more informative than 5,000 drawn from the same distribution.
- How do you run evals in CI without the cost getting out of hand?
- Tier them. Deterministic tests and assertion evals against recorded fixtures run on every commit with no model calls at all. A stratified subset runs on pull requests that touch prompts, tools or retrieval. The full suite with both judges runs nightly and before release. Most teams that abandon evals did so because they put the full suite on every commit.