Eliminate Flaky Tests by Removing Runtime Model Calls

Flaky tests caused by a runtime model call

Your CI pipeline fails intermittently. The same commit passes on one runner, fails on another. The stack trace points to a validation step that invokes a language model over HTTP. The model returns a different suggestion each time. The test suite treats the suggestion as the ground truth, so the assertion sometimes passes, sometimes does not.

The failure mode is clear: a runtime dependency on an external model injects non‑determinism. The build no longer reflects the code alone; it reflects the state of a service you do not own. The symptom is flaky tests, but the underlying risk is larger – a production release may behave differently tomorrow when the model is updated.

How a model call corrupts deterministic software

Deterministic software produces the same output given the same input, regardless of when or where it runs. A model call violates every invariant that deterministic builds rely on.

• Clock drift – the request includes a timestamp. Different runners have slightly different clocks, producing different request signatures.
• Randomness – many model APIs expose a temperature or seed parameter. If the client does not fix the seed, the model samples a different token sequence on each call.
• Network order – parallel test workers may race to send requests. The order in which the model processes them can affect internal state, especially for models that keep a short‑term cache.

Because the model’s behaviour is not owned, you cannot guarantee repeatability. The build artefact is effectively a hybrid of your code and a moving target. That hybrid is the source of AI hallucination risk: the model may fabricate a response that passes a test today but is later deemed incorrect.

Replace the call with owned logic

The engineering posture that removes the variance is to own the logic outright. Instead of asking a model at build time, extract the required knowledge into a static artefact and treat it as part of your source tree.

Consider a function that validates user‑generated SQL against a whitelist of safe patterns. The original implementation queried a model:

async function isSafe(sql: string): Promise<boolean> {
  const resp = await fetch('https://model.api/validate', {
    method: 'POST',
    body: JSON.stringify({ query: sql })
  });
  const { safe } = await resp.json();
  return safe;
}

The function’s result changes whenever the model’s training data is refreshed. To make it deterministic, run the model once during a controlled evaluation, capture the whitelist, and embed it:

// generated at evaluation time, checked into repo
const SAFE_PATTERNS = [
  /^SELECT \* FROM users WHERE id = \d+$/,
  /^INSERT INTO orders\(.*\) VALUES\(.*\)$/,
];

function isSafe(sql: string): boolean {
  return SAFE_PATTERNS.some(p => p.test(sql));
}

The generation step is a one‑off operation. It can be performed with the SHPBL library that harvests reusable capability from existing repositories. The library is model‑independent; the artefact it produces is a plain file, repeatable byte‑for‑byte, and its checksum is published in the root of trust. Once the static list lives in version control, every build sees the same data, and the test suite becomes stable.

The shift also removes the hallucination surface. The model no longer runs at runtime, so it cannot fabricate an answer that passes a test but fails in production. The only remaining risk is the correctness of the generated list, which can be verified once with the certification harness. The harness will label the artefact CERTIFIED if it passes its contract, or PROVISIONAL if reproducibility is proven but correctness is still under review. In either case, the build is now repeatable.

A concrete step for Monday

1. Identify a single runtime model call that influences a test or build artefact.
2. Run the call in an isolated evaluation environment and capture the output.
3. Store the captured output as a static file in the repository, and replace the call with a pure function that reads the file.

Commit the change, run the full CI suite, and verify that the flaky failures disappear. The next build will be deterministic, and you will have eliminated a source of AI hallucination risk without altering any business logic.

By treating the model output as data to be owned rather than a service to be called, you align your pipeline with deterministic software principles and achieve reproducible builds. The approach mirrors the argument in the pillar post on model independence, and it can be applied incrementally across any codebase that currently leans on runtime model inference.