How to Test AI Systems Without Chasing Flaky Tests

Run the same prompt twice and you can get two different answers. Exact-match assertions break on the first synonym. The test goes red, someone marks it skipped, and the suite quietly stops meaning anything. Testing an AI system well means never reaching that point.

What counts as an AI system

Start with what you are testing, because it is not the model.

An AI system is the code you ship around a model call. Trace one request through a document Q&A feature and it moves through prompt assembly, which stitches a template together with the user’s question and the conversation history. Then retrieval, which pulls candidate documents out of an index. Then the model call itself. Then any tool the model invokes back into your code. Then a parser that turns the response into a typed object, and the error handling that decides what happens when a link in that chain breaks.

You wrote every one of those parts except the model. The provider tests the model. The chain around it is yours, and that is where your bugs live.

That boundary sets the scope here. This guide covers systems built on a model you did not train: retrieval-augmented Q&A, document extraction, classification, chat, and agents that call tools. It does not cover training or benchmarking a model of your own, which is separate work with separate instruments. An MMLU score says nothing about the bug in your chunking code.

Why testing an AI system breaks an ordinary suite

Two properties do the damage. The same input does not produce the same output, so a byte-for-byte assertion fails on a harmless rewording. And across part of the output no single correct answer exists, so there is nothing to compare against even in principle.

Advice on testing AI systems usually opens with semantic similarity scoring. That skips a step. Most of an AI feature is ordinary software: retrieval, schema, routing, tool calls, cost, latency. That part is deterministic and deserves exact assertions. Only a thin slice is genuinely open-ended.

This guide works in that order. Pin what you can control. Sample what you cannot. Gate CI on a pass rate with a confidence bound, not on one lucky run. At the end you have a suite that fails for a reason, so nobody mutes it.

What you need first

• A test runner. The examples here run on pytest 9.1, but the ideas port to Vitest or JUnit.
• API access to the model your product calls, plus budget for repeat calls. Sampling costs money.
• A frozen fixture set: 20 to 50 real inputs with known-good outputs.
• A pinned retrieval corpus, if the feature retrieves anything.

The fixture set is the part teams skip. Skip it and every later step measures noise.

Step 1: Split the surface in two

Sort every check into one of two buckets before you write code.

Most teams load up the right column. That is the mistake. A wrong tool call, a dropped field, and a 4x cost regression are all exact failures, and none of them need a scorer.

Anything in the left column is a normal test. Write it as one. It runs in milliseconds, costs nothing, and fails for a legible reason.

This split also replaces the three-layer framing (data, model, business impact) that the vendor blogs push. Those three layers are an org chart. They tell you who cares about a failure. They do not tell you what to assert.

Step 2: Pin what the provider lets you pin

Set temperature to 0. Pass a fixed seed if your provider offers one. Name an exact model snapshot rather than a floating alias. Freeze the corpus. Version the system prompt.

Then expect all of it to leak.

Anthropic’s API reference is blunt about it. On the temperature parameter: “Note that even with temperature of 0.0, the results will not be fully deterministic.” OpenAI’s cookbook says its system makes “a best effort to sample deterministically” with a seed, that “determinism is not guaranteed,” and that you should watch system_fingerprint for backend changes.

The cause is not mysterious. Thinking Machines Lab traced it to batch invariance: inference kernels give slightly different numbers depending on batch size, and batch size moves with server load. Their batch-invariant kernel library reports the gap directly. Out of 1000 completions of length 100, they saw 18 unique samples. With batch-invariant kernels, one.

So pinning narrows variance. It does not remove it. Your batch size depends on other people’s traffic, which means a byte-for-byte assertion can go red on a Tuesday for reasons inside someone else’s request.

One pin does hold. Schema-constrained decoding moves format checks out of your test suite and into the API contract. Anthropic’s strict tool use adds strict: true to a tool definition “to ensure Claude’s tool calls always match your schema exactly.” Turn that on and a whole class of tests disappears.

Step 3: Assert a pass rate, not an answer

For the right-hand column, run the same input many times and score the fraction that passes. Then check a confidence bound against a floor, so a small sample cannot pass on luck.

# tests/test_extraction.py
# extract_invoice() is your own call into the model; INVOICE is one fixture.
import math

SAMPLES = 50
MIN_PASS_RATE = 0.90


def wilson_lower_bound(successes: int, trials: int, z: float = 1.96) -> float:
    """Lower bound of the 95% confidence interval for a pass rate."""
    if trials == 0:
        return 0.0
    p = successes / trials
    denom = 1 + z**2 / trials
    centre = p + z**2 / (2 * trials)
    margin = z * math.sqrt(p * (1 - p) / trials + z**2 / (4 * trials**2))
    return (centre - margin) / denom


def test_invoice_total_is_extracted():
    passes = sum(
        1 for _ in range(SAMPLES)
        if extract_invoice(INVOICE)["total_cents"] == 128_400
    )
    lower = wilson_lower_bound(passes, SAMPLES)
    assert lower >= MIN_PASS_RATE, (
        f"{passes}/{SAMPLES} correct; 95% lower bound {lower:.3f} "
        f"is under the {MIN_PASS_RATE} floor"
    )

Now the part that surprises people. Sample size decides what you are allowed to claim, and small samples claim very little. Here is the Wilson lower bound for a run where every sample passed:

A flawless 20-for-20 does not support a 90% claim. You need 35 perfect runs to clear a 0.90 floor, 73 to clear 0.95, and 381 to clear 0.99. Pick the floor with that price in view.

The bound is strict about near-misses too. At 50 samples, 49 passes lands at 0.895 and fails a 0.90 gate. That is the point: one bad sample out of 50 is evidence, not noise to re-run away.

Step 4: Write metamorphic relations for the rest

Some outputs have no known-correct answer to compare against. Test the relationship between runs instead. Feed the system inputs that differ in a way that should not change the verdict, then assert that the verdicts agree.

from collections import Counter

PARAPHRASES = [
    "What is the refund window for a damaged item?",
    "How long do I have to return something that arrived broken?",
    "If my order shows up damaged, by when must I ask for a refund?",
]


def test_paraphrases_reach_the_same_policy():
    answers = [classify_policy(q) for q in PARAPHRASES]
    label, count = Counter(answers).most_common(1)[0]
    assert count == len(answers), f"paraphrases disagreed: {answers}"

Three relations carry most of the value:

• Paraphrase invariance. Reword the question, keep the answer.
• Order invariance. Shuffle retrieved documents or list items. A ranking may move; a classification should not.
• Irrelevance invariance. Append a sentence about the weather. The answer should ignore it.

These tests fail loudly and cheaply, and they need no labels.

Step 5: Bring in a judge model last, and grade the judge

Use a model as a grader only for what resists code-based grading: faithfulness to a source, tone, whether an answer addressed the question at all. Anthropic’s eval guidance puts automation first and says to “prioritize volume over quality,” because more questions with automated grading beat a handful of hand-graded ones. It also advises using a different model to grade than the one that generated the output.

A judge is an instrument, and instruments have error. The MT-Bench authors report in the FastChat repository that “humans and GPT-4 judge achieve over 80% agreement, the same level of agreement between humans,” measured across 80 questions with 3.3K human annotations. Read that in both directions. A judge can track human preference about as well as another human. It also disagrees with a human roughly one time in five.

So measure your judge before you trust it. Hand-label 50 fixtures, run the judge over them, and record the agreement rate. Track that number as its own metric. Change the judge prompt and that number moves.

Two open-source runners handle the plumbing. DeepEval is pytest-native: you build an LLMTestCase, attach a metric such as GEval with a threshold, and run deepeval test run test_chatbot.py. promptfoo is config-first and mixes deterministic and graded checks in one file:

tests:
  - vars:
      language: French
      input: Hello world
    assert:
      - type: contains-json
      - type: javascript
        value: output.toLowerCase().includes('bonjour')
      - type: similar
        value: was geht
        threshold: 0.6

Note the shape of that block. Two cheap deterministic checks, one scored check with an explicit threshold. That ratio is the goal.

Step 6: Gate CI against the last green baseline

An absolute score is the wrong gate. Model providers ship updates, and your prompt changes weekly, so a fixed 0.92 will either block every release or catch nothing.

Store the pass rate from the last green build. Fail when the new rate drops below it by more than the sampling noise you measured in step 3. A drop from 0.94 to 0.91 across 200 samples is a regression worth blocking. The same drop across 20 samples is nothing.

Split the schedule to match the cost. Deterministic checks run per commit, in seconds. Sampled checks run nightly, on the full fixture set, with the result written to a file the next run compares against.

Common pitfalls

Muting instead of widening. The moment a probabilistic test gets skip, the feature is untested. Widen the tolerance and record why, or delete the test outright. A skipped test is worse than no test, because it still looks like coverage.

One sample and a green build. Running an open-ended check once tells you the pass rate is somewhere above zero. Nothing more.

Grading with the model that generated. Cheap, convenient, and biased toward its own output. Use a different model, per Anthropic’s guidance above.

Testing the model instead of the system. Your users do not hit the model. They hit retrieval, prompt assembly, tool calls, parsing, and error handling. Benchmark scores tell you nothing about the bug in your chunking code.

Fixtures that rot. A fixture set built in March describes March’s product. Re-check it quarterly against real traffic, and add every production incident to it as a case.

Floating model aliases. An alias that silently rolls to a new version turns every regression into a mystery. Pin the snapshot and upgrade on purpose.

Build the fixture set faster

Steps 3 and 5 both need volume: dozens of realistic inputs, with relationships that hold across tables. Hand-writing them is the slowest part of the whole setup. The relational test data generator produces linked users, orders, and line items as JSON or SQL, which gives you a seeded corpus to point the retrieval and extraction tests at.

Related reading

• Testing in the Age of AI: The Contract-Driven Loop sets out the contract, action, and artifact loop this guide implements.
• Using AI Agents for Testing covers the other direction: agents writing your tests.
• Will AI Replace QA Testers? on where the judgment work moves.
• Relational Test Data Generator for the fixture corpus.