Test evidence

How we measure #

Prefix caching is the real lever #

Providers cache prompt prefixes. The question was how much of a realistic prompt comes back cached, and what breaks it. One stable system prompt of roughly 860 tokens, called four ways:

The fourth row is the whole finding. A single volatile value at the front of a prompt costs you the entire cached prefix on every subsequent call. So the gateway canonicalises the cacheable prefix — sorted keys, compact JSON, per-request values moved out of it — prices cached tokens at your contract's cached rate, and puts the hit rate on a tile so a regression is visible the day it happens.

On a mixed session workload rather than a synthetic repeat, 78% of prompt tokens came back cached.

History windowing lost to it #

Trimming old turns out of a conversation is the classic token-saving move, and it was the headline of our own earlier material. Measured on a ten-turn conversation, with trimming off and then on:

Trimming removed 2,163 real tokens and destroyed the cached prefix, dropping it from 5,088 tokens to 80. The trimmed request has fewer tokens and more of them at full price. It only wins at a cached discount below roughly 43%, which no provider offers.

A local model as the small tier #

A 4-billion-parameter model running on a laptop, declared as a small-tier catalog entry, against the Factory's small tier. Five uncached calls each, measured end to end through the full gateway chain, on a task both answered correctly:

Roughly an eighth of the latency on routine work, and it loses clearly on hard work. That is exactly why it is a small-tier entry a caller opts into with iag/fast, and not a cache placed in front of everything. The honest value here is latency and locality: upstream spend on routine work was already small.

The outage cycle #

Exercised end to end against the running stack:

Three classifiers, none of which earned its place #

The obvious feature is a router that reads each request and decides whether it needs the expensive model. We tried three: a 4-billion-parameter local model as a judge, a published task-and-complexity encoder, and an open-source router trained on human preference data. Each was scored on the only question that matters operationally — will the cheap tier get this right — against deterministic checkers.

Results for the strongest of the three, on 32 prompts built so the cheap tier had something to fail:

Every routing threshold cost more and was no more correct. The cause is not the routers. On this catalog and this traffic the expensive tier is simply not more correct than the cheap one: on the hard half of the set the small tier scored 88% and the medium tier 81%. A router can only convert cost into quality where a quality gap exists to buy.

There is also a boundary problem worth stating: the two strongest routers in that open-source package call a third-party embedding API for every prompt. On a gateway whose purpose is to keep prompt content on a governed path, that is disqualifying regardless of accuracy.

Tool memoisation, stated precisely #

Serving a repeated read-only tool call from what the conversation already answered saves the client one tool execution and one full round-trip of wall-clock latency per repeat. It does not save upstream tokens: the continuation is an inference the client would have made anyway, and both inferences are metered.

Its value therefore depends entirely on the repeat rate of read-only calls in your own agent traces — which the evidence rows make queryable, so you can measure it rather than take a number from us.

What pre-merge review caught #

Security findings are evidence too. The most recent change to the request path — letting a catalog model declare its own upstream — went through an adversarial review before merge. Six defects were found and fixed in the same change, each with a regression test that fails without the fix.

Results that went against us #

Kept here on purpose, because a page of only favourable results is not evidence.

• Our headline token-saving feature lost to the provider's cache. History windowing is no longer a savings claim.
• Our own "tokens saved" counter was overstated. It is a characters-per-token estimate, it was reporting more than the upstream's counters supported, and it is now labelled an estimate and excluded from every claim.
• Three attempts at automatic routing all failed to beat the simplest policy. The feature was not shipped; the decision was moved to the caller.
• The local model is not a general win. It matches on routine work and loses clearly on hard work, which is why it is opt-in and tier-scoped.

Reproduce it on your own traffic #

Nothing here needs our data. Point the gateway at your Factory and read your own counters.

# prefix cache: same prompt twice, watch the cached share
for i in 1 2; do
  curl -si localhost:8080/v1/chat/completions -H "Authorization: Bearer $KEY" \
    -d '{"model":"iag/fast","messages":[{"role":"system","content":"<your real system prompt>"},
                                        {"role":"user","content":"hello"}]}' \
  | grep -i 'x-iag-upstream-cached-tokens\|x-iag-cost-usd'
done

# the same prompt with a timestamp in front — the cached share should collapse
# then check the aggregate over a day, from the admin API
curl -s -H "$A" "$B/overview?workspace=ws-pilot&window=24h" \
  | jq '{cache_hit_rate, upstream_prefix_hit_rate, spend_usd, savings_usd}'

# small tier versus large tier on YOUR tasks, scored by your own checkers:
# run each prompt through both, compare correctness, then compare cost per 1k.
curl -s -H "$A" "$B/requests?workspace=ws-pilot&model=openai/gpt-oss-20b" | jq '.data[].cost_usd'