Knowing when not to ask a model

A red test suite asks one question before any other: what kind of failure is this, and does it need a human at all? Answering that with a frontier model is easy, slow and expensive. Answering it with an if is none of those things — when an if is enough.

So triage is a ladder, and each rung has to justify skipping the one below it.

1. Tier 0

   Assertions, XML parse, a known-failure catalogue

   Deterministic. Always on. No egress.

   free · instant

2. Tier 1

   Jev — typed decisions with calibrated confidence

   Probabilistic. Opt-in behind a flag and an env key. Third party.

   70–500 ms · ~450 input tokens

   Below this line, nothing leaves the machine

3. Tier 2

   Mid-tier LLM

   Probabilistic. Escalation only, when Tier 1 is unsure.

   provider

4. Tier 3

   Frontier LLM

   Probabilistic. Novel failures only — the ones nothing below can name.

   provider · most expensive

The governing rule is short: if a rule can be written as an if, it stays in Tier 0 and never reaches a model.

Tier 0 is three layers, not one

Most of the work happens before anything probabilistic runs, and it is worth naming because “deterministic checks” undersells it:

• 0a — known test names. A catalogue of signatures, each with a hint and a route.
• 0b — known assertion messages, matched as substrings of the XML <message>. Every entry names the committed Assert literal it came from, so an entry cannot drift away from the code that produces it.
• 0c — subsystem extraction from the class name inside the test’s fullname. Best-effort, and labelled as such.

The two tables fire independently, which sounds obvious and is not: it means a failure whose name nobody has seen before can still be classified by its message. A self-test proves it by emptying one table and checking the other still matches — so the independence cannot quietly rot.

The failure this exists to explain

Down-stack integration tests call Assert.Ignore. That lands them in the results XML as Skipped, not Failed — so a triage pass that scans failed cases never sees them at all. The suite is green-ish, a whole layer did not run, and nothing says so.

Silent absence is harder to notice than a red mark, so the tool reports the skipped set explicitly and flags the integration ones with the docker compose up -d hint they almost always need. Of everything here, this is the part that has saved the most time, and it never touches a model.

What Tier 1 actually is

Tier 1 is Jev, TypeSafe AI’s first System One model — POST /v1/systemone, version jev-1.13.0, early access, evaluated 2026-09-21. Every finding below belongs to that build on that date.

It is not an LLM and not a validator. It takes a state — natural-language text — plus a set of typed questions, and returns typed decisions with calibrated confidence in 70–500 ms, at $0.042 per million input tokens with free output. A triage call is about 450 input tokens.

Three primitives, all evaluated against one state in a single round trip:

Primitive	Returns
Noul	A yes/no probability
Choice	One option from a closed set, plus a probability for each
Score	A number against a written rubric

“Type-safe” means it cannot return a malformed object or an undeclared option. It does not mean the answer is correct — it can return a valid-but-wrong choice with high confidence. The confidence is the escape hatch.

What it is actually asked

Two questions per failure. A failure_class Choice over a closed set — infra, assertion, art_invariant, flaky, novel — and an environmental Noul: is this the local machine rather than the code?

The routing rule is where the judgement lives:

• Act on the class only at confidence ≥ 0.75, and only when it is not novel. Anything else escalates.
• An environmental probability above 0.8 overrides the class entirely and routes to the infra step — because “your Docker isn’t up” outranks any opinion about which assertion broke.

The findings, as measured

The live path was exercised against the real API on 2026-09-21. What it showed:

• The same input was classified two different ways — art_invariant at 0.35 confidence on one run, assertion at 0.36 on another. The route flips between “regenerate the art” and “look at a constant”. Tier 1 is not deterministic, and anything built on it must treat that as a given rather than an anomaly. A Tier 1 answer is a hint, not a verdict.
• One question over two concatenated failures came back at 0.17 confidence. Asked separately, the same failures scored 0.35–0.45, and a clear-cut case scored 0.94. That is the shape you want — honest low confidence on ambiguity — but only if the question is scoped to one failure. Hence one call per failure rather than a fan-out over a shared state.
• The first live call returned 422: the score primitive takes criteria, not min/max. The fix was not to spell the schema out correctly — it was to delete the score question and use the Choice’s own confidence, which every Choice answer already carries. Fewer fields, and no invented schema to keep in step with someone else’s API.
• The confidence floor is set, not yet earned. It rests on a handful of observations, not a calibration run — and the one high-confidence sample was an infra case Tier 0 already catches, so the floor has never been exercised at a boundary where Jev would actually route rather than escalate. Saying so is the point: an unvalidated threshold presented as a tuned one is how a probabilistic component quietly becomes trusted.
• “Never in CI” is a convention with no guard. It is documented and observed, and nothing enforces it — it rests on the env-key requirement making a keyless CI job fail loudly, not on a rule. A convention without an assertion is a convention with a half-life.

One finding is about the catalogue rather than the model, and it is the one I would keep if I could keep only one. An entry was briefly added by copying a string out of a gotchas document. It was unreachable — that string can never appear in a results XML, because it lives in a compose file and the tests that would surface it skip. It was deleted rather than shipped. A catalogue entry that cannot fire is not harmless; it is decoration that makes the catalogue look more complete than it is.

Egress is the real design constraint

An assertion message can carry a device id, a user id or a payload fragment. So redaction of JWT-, GUID- and long-token-shaped runs happens before anything leaves the machine, is mandatory, and cannot be disabled from the command line.

The long-token pattern deliberately excludes _, so a long test method name is never blanked — redaction that eats the evidence is its own kind of failure.

Everything above Tier 0 is opt-in behind a flag, requires an environment key, and degrades loudly: a missing key is a usage error and exits 2, a network failure falls back to Tier 0 and says that it did, paid calls are capped at five by default, and 429/529 retry with backoff. Triage is advisory throughout — it exits 0 on a red run, and the build gate stays deterministic.

Could a smaller model do Tier 1?

Unbuilt, and worth stating as a hypothesis rather than a plan.

Tier 1’s entire job is one closed-set classification and one yes/no, over ~450 tokens of already-redacted text. That is a small-model task by any reasonable measure. Jev supplies two things a raw model does not — a schema that cannot return an undeclared option, and a calibrated confidence number.

The schema guarantee is now commodity: constrained decoding gives the same property locally, moving the guarantee from the vendor to the decoder. The confidence number is the hard part, and it is load-bearing here — the 0.75 floor is the whole safety mechanism. But that floor is already unearned with Jev, so the honest position is that neither option has earned it, and a local swap would have to earn it rather than inherit it.

What a local model wins is this page’s own constraint. Redaction is mandatory because assertion text can carry identifiers. A local model does not mitigate that risk — it deletes it. No egress, no key custody, no paid-call cap, no backoff, no dependency on an early-access product. And the determinism may invert: the measured complaint against Tier 1 is that one input came back two ways, which is a property of sampling. A local model pinned at temperature zero could be the more reproducible of the two.

The test that would settle it, stated so it can be run: replay the captured red runs through both over the same redacted states, and compare three things — agreement with the human label, run-to-run stability across repeats, and whether the confidence number separates right answers from wrong ones at all. Only the third makes a floor mean anything. Without it, a floor is decoration.

The same idea one layer up — classify a task as trivial, standard or complex and route it to a cheap or an expensive model, with the classifier seeing task text and never file bodies — is the unbuilt third position on the workshop page.

Why this is the interesting part

The tempting version of this project is “we added AI to test triage”. The useful version is a ladder where three of the four tiers are cheaper, faster and more predictable than the one everybody talks about, and the model is confined to the one question the deterministic layer genuinely cannot answer.